forked from roengram/hadoop-release
-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathHDP-CHANGES.txt
218 lines (215 loc) · 21.2 KB
/
HDP-CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
Changes in HDP and not in Apache Hadoop-3.1.1 release
=========================================================================
#### (NOTE - This is -generated from git-log, don't add entries to this file directly - see BUG-112676 for instructions.) ###
BUG-115218. HADOOP-15959: Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple.
HDFS-14112. Avoid recursive call to external authorizer for getContentSummary.
HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances.
HADOOP-14445. Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. Contributed by Daryn Sharp, Xiao Chen, Rushabh S Shah.
MAPREDUCE-7158. Inefficient Flush Logic in JobHistory EventWriter. (Zichen Sun via wangda)
YARN-9002. Improve keytab loading for YARN Service. Contributed by Gour Saha
HADOOP-15872. ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
HADOOP-15815. Upgrade Eclipse Jetty version to 9.3.24. Cotributed by Boris Vulikh.
HADOOP-15846. ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries.
HADOOP-15812. ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console.
BUG-114054. YARN-8950. Fix compilation issue due to dependency convergence error for hbase.profile=2.0.
HDFS-14026. Overload BlockPoolTokenSecretManager.checkAccess to make storageId and storageType optional. Contributed by Arpit Agarwal.
HDFS-13941. make storageId in BlockPoolTokenSecretManager.checkAccess optional. Contributed by Wei-Chiu Chuang.
BUG-112221. YARN-8854. Upgrade jquery datatable version references to v1.10.19. Contributed by Akhil PB.
BUG-112407. HADOOP-15837. DynamoDB table Update can fail S3A FS init. Contributed by Steve Loughran.
YARN-8858. CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used. Contributed by Wangda Tan.
YARN-8804. resourceLimits may be wrongly calculated when leaf-queue is blocked in cluster with 3+ level queues. Contributed by Tao Yang
YARN-8771. CapacityScheduler fails to unreserve when cluster resource contains empty resource type. Contributed by Tao Yang.
YARN-8868. Set HTTPOnly attribute to Cookie
HADOOP-15882. Upgrade maven-shade-plugin from 2.4.3 to 3.2.0. Contributed by Takanobu Asanuma.
YARN-4606. CapacityScheduler: applications could get starved because computation of #activeUsers considers pending apps. Contributed by Manikandan R
HADOOP-15823. ABFS: Stop requiring client ID and tenant ID for MSI (Contributed by Da Zhou via Daniel Templeton)
HADOOP-15839. Review + update cloud store sensitive keys in hadoop.security.sensitive-config-keys. Contributed by Steve Loughran.
YARN-8535. Fix DistributedShell unit tests. Contributed by Abhishek Modi.
HDFS-13662. TestBlockReaderLocal#testStatisticsForErasureCodingRead is flaky
HADOOP-15850 CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0
HADOOP-15851. Disable wildfly logs to the console. Contributed by Vishwajeet Dusane.
BUG-112690. HADOOP-15825. ABFS: Enable some tests for namespace not enabled account using OAuth. Contributed by Da Zhou.
BUG-112689. HADOOP-15809. ABFS: better exception handling when making getAccessToken call. Contributed by Da Zhou
BUG-112086. Fixed Authentication Header passing.
BUG-111586. YARN-8758. Support getting PreemptionMessage when using AMRMClientAsyn. (Zian Chen via wangda)
HADOOP-15680. ITestNativeAzureFileSystemConcurrencyLive times out. Contributed by Andras Bokor.
HADOOP-15795. Make HTTPS the default protocol for ABFS. Contributed by Da Zhou.
HADOOP-15792. typo in AzureBlobFileSystem.getIsNamespaceEnabeld. Contributed by Abhishek Modi.
HADOOP-15793. ABFS: Skip unsupported test cases when non namespace enabled in ITestAzureBlobFileSystemAuthorization Contributed by Yuan Gao.
HADOOP-15801. ABFS: Fixing skipUserGroupMetadata in AzureBlobFileSystemStore. Contributed by Da Zhou
HADOOP-15723. ABFS: Ranger Support. Contributed by Yuan Gao.
HADOOP-15739. ABFS: remove unused maven dependencies and add used undeclared dependencies. Contributed by Da Zhou.
BUG-111853. MAPREDUCE-7125. JobResourceUploader creates LocalFileSystem when it's not necessary. (Peter Cseh via wangda)
HADOOP-15778. ABFS: Fix client side throttling for read. Contributed by Sneha Varma.
HADOOP-15704. Mark ABFS extension package and interfaces as LimitedPrivate/Unstable. Contributed by Steve Loughran.
HADOOP-15773. Fixing checkstyle and other issues raised by Yetus.
HADOOP-15769. ABFS: distcp tests are always skipped. Contributed by Steve Loughran
HADOOP-15719. Fail-fast when using OAuth over http. Contributed by Da Zhou.
HADOOP-15714. Tune abfs/wasb parallel and sequential test execution. Contributed by Da Zhou.
HADOOP-15715. ITestAzureBlobFileSystemE2E timing out with non-scale timeout of 10 min. Contributed by Da Zhou
HADOOP-15744. AbstractContractAppendTest fails against HDFS on HADOOP-15407 branch. Contributed by Steve Loughran.
HADOOP-15694. ABFS: Allow OAuth credentials to not be tied to accounts. Contributed by Sean Mackrory.
HADOOP-15740. ABFS: Check variable names during initialization of AbfsClientThrottlingIntercept. Contributed by Sneha Varma.
HADOOP-15745. ABFS: Add ABFS configuration to ConfigRedactor.
Fixing findbugs and license issues related to: HADOOP-15703. ABFS - Implement client-side throttling. Contributed by Sneha Varma and Thomas Marquardt.
BUG-111485. YARN-8720. CapacityScheduler does not enforce max resource allocation check at queue level. Contributed by Tarun Parimi.
BUG-111427. HADOOP-15748. S3 listing inconsistency can raise NPE in globber. Contributed by Steve Loughran.
HADOOP-15728. ABFS: Add backward compatibility to handle Unsupported Operation for storage account with no namespace feature.
HADOOP-15753. ABFS: support path "abfs://mycluster/file/path" Contributed by Da Zhou.
HADOOP-15757. ABFS: remove dependency on common-codec Base64. Contributed by Da Zhou.
BUG-110754. HADOOP-15677. WASB: Add support for StreamCapabilities. Contributed by Thomas Marquardt.
HADOOP-15703. ABFS - Implement client-side throttling. Contributed by Sneha Varma and Thomas Marquardt.
HADOOP-15664. ABFS: Reduce test run time via parallelization and grouping. Contributed by Da Zhou.
HADOOP-15663. ABFS: Simplify configuration. Contributed by Da Zhou.
HADOOP-15692. ABFS: extensible support for custom oauth. Contributed by Junhua Gu and Rajeev Bansal.
HADOOP-15682. ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle. Contributed by Thomas Marquardt.
HADOOP 15688. ABFS: InputStream wrapped in FSDataInputStream twice. Contributed by Sean Mackrory.
BUG-110124.YARN-8726. [UI2] YARN UI2 is not accessible when config.env file failed to load. Contributed by Akhil PB.
[BUG-109479] [YARN-8675] Setting hostname of docker container breaks with host networking mode for Apps which do not run as a YARN service
HADOOP-15596. Stack trace should not be printed out when running hadoop key commands. Contributed by Kitti Nanasi.
HADOOP-15550. Avoid static initialization of ObjectMappers
HDFS-13690. Improve error message when creating encryption zone while KMS is unreachable. Contributed by Kitti Nanasi.
HADOOP-15316. GenericTestUtils can exceed maxSleepTime. Contributed by Adam Antal.
HDFS-13485. DataNode WebHDFS endpoint throws NPE. Contributed by Siyao Meng.
HDFS-13663. Should throw exception when incorrect block size is set. Contributed by Shweta.
HADOOP-15549. Upgrade to commons-configuration 2.1 regresses task CPU consumption
HADOOP-15581. Set default jetty log level to INFO in KMS. Contributed by Kitti Nanasi.
HDFS-13622. mkdir should print the parent directory in the error message when parent directories do not exist. Contributed by Shweta.
HADOOP-15395. DefaultImpersonationProvider fails to parse proxy user config if username has . in it. Contributed by Ajay Kumar.
HADOOP-14396. Add builder interface to FileContext. Contributed by Lei (Eddy) Xu.
HDFS-11060. make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurable. Contributed by Lantao Jin.
HADOOP-15611. Log more details for FairCallQueue. Contributed by Ryan Wu.
HADOOP-15591. KMSClientProvider should log KMS DT acquisition at INFO level. Contributed by Kitti Nanasi.
HADOOP-14313. Replace/improve Hadoop's byte[] comparator. Contributed by Vikas Vishwakarma.
HADOOP-15307. NFS: flavor AUTH_SYS should use VerifierNone. Contributed by Gabor Bota.
HADOOP-15520. Add tests for various org.apache.hadoop.util classes. Contributed by Arash Nabili
HDFS-13642. Creating a file with block size smaller than EC policy's cell size should fail.
HADOOP-15512. Clean up Shell from JDK7 workarounds. Contributed by Zsolt Venczel.
HADOOP-14783. [KMS] Add missing configuration properties into kms-default.xml. Contributed by Chetna Chaudhari.
HDFS-13646. DFSAdmin doesn't display specialized help for triggerBlockReport. Contributed by Takanobu Asanuma.
HDFS-13440. Log HDFS file name when client fails to connect. Contributed by Gabor Bota.
HDFS-13626. Fix incorrect username when deny the setOwner operation. Contributed by Zsolt Venczel.
HADOOP-15455. Incorrect debug message in KMSACL#hasAccess. Contributed by Yuen-Kuei Hsueh.
HDFS-13598. Reduce unnecessary byte-to-string transform operation in INodesInPath#toString. Contributed by Gabor Bota.
HDFS-13573. Javadoc for BlockPlacementPolicyDefault is inaccurate. Contributed by Zsolt Venczel.
HADOOP-15377. Improve debug messages in MetricsConfig.java
HDFS-13468. Add erasure coding metrics into ReadStatistics. (Contributed by Lei (Eddy) Xu)
HADOOP-15402. Prevent double logout of UGI's LoginContext
HADOOP-12502 SetReplication OutOfMemoryError. Contributed by Vinayakumar B.
HADOOP-15340. Provide meaningful RPC server name for RpcMetrics. Contributed by Elek Marton.
HDFS-13363. Record file path when FSDirAclOp throws AclException. Contributed by Gabor Bota.
HDFS-13292. Crypto command should give proper exception when trying to set key on existing EZ directory. Contributed by Ranith Sardar.
HADOOP-14855. Hadoop scripts may errantly believe a daemon is still running, preventing it from starting. Contributed by Robert Kanter.
HDFS-13357. Improve AclException message "Invalid ACL: only directories may have a default ACL.". Contributed by Gabor Bota.
HDFS-13357. Improve AclException message "Invalid ACL: only directories may have a default ACL.". Contributed by Gabor Bota.
HADOOP-15679. ShutdownHookManager shutdown time needs to be configurable & extended. Contributed by Steve Loughran.
HDFS-13772. Erasure coding: Unnecessary NameNode Logs displaying for Enabling/Disabling Erasure coding policies which are already enabled/disabled. Contributed by Ayush Saxena
HADOOP-9214. Create a new touch command to allow modifying atime and mtime. Contributed by Hrishikesh Gadre.
HADOOP-15655. Enhance KMS client retry behavior. Contributed by Kitti Nanasi.
HDFS-10240. Race between close/recoverLease leads to missing block. Contributed by Jinglun, zhouyingchao and Wei-Chiu Chuang.
HDFS-13732. ECAdmin should print the policy name when an EC policy is set. Contributed by Zsolt Venczel.
HDFS-13217. Audit log all EC policy names during addErasureCodingPolicies. Contributed by liaoyuxiangqin.
HDFS-13819. TestDirectoryScanner#testDirectoryScannerInFederatedCluster is flaky
HDFS-13668. FSPermissionChecker may throws AIOOE when check inode permission. Contributed by He Xiaoqiao.
HDFS-13658. Expose HighestPriorityLowRedundancy blocks statistics. Contributed by Kitti Nanasi.
HDFS-13447. Fix Typos - Node Not Chosen. Contributed by Beluga Behr.
HDFS-13785. EC: 'removePolicy' is not working for built-in/system Erasure Code policies. Contributed by Ayush Saxena
HDFS-13799. TestEditLogTailer#testTriggersLogRollsForAllStandbyNN fails due to missing synchronization between rollEditsRpcExecutor and tailerThread shutdown. Contributed by Hrishikesh Gadre.
HDFS-13792. Fix FSN read/write lock metrics name. Contributed by Chao Sun.
HADOOP-14212. Addendum patch: Expose SecurityEnabled boolean field in JMX for other services besides NameNode. Contributed by Adam Antal.
HADOOP-14212. Expose SecurityEnabled boolean field in JMX for other services besides NameNode. Contributed by Adam Antal.
HADOOP-15661. ABFS: Add support for ACL. Contributed by Junhua Gu and Da Zhou.
HADOOP-15660. ABFS: Add support for OAuth Contributed by Da Zhou, Rajeev Bansal, and Junhua Gu.
HADOOP-15669. ABFS: Improve HTTPS Performance. Contributed by Vishwajeet Dusane.
HADOOP-15659. Code changes for bug fix and new tests. Contributed by Da Zhou.
HADOOP-15446. ABFS: tune imports & javadocs; stabilise tests. Contributed by Steve Loughran and Da Zhou.
HDFS-13750. RBF: Router ID in RouterRpcClient is always null. Contributed by Takanobu Asanuma.
YARN-8679. [ATSv2] If HBase cluster is down for long time, high chances that NM ContainerManager dispatcher get blocked. Contributed by Wangda Tan.
HADOOP-15674. Test failure TestSSLHttpServer.testExcludedCiphers with TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite. Contributed by Szilard Nemeth.
HDFS-13747. Statistic for list_located_status is incremented incorrectly by listStatusIterator. Contributed by Antal Mihalyi.
YARN-8612. Fix NM Collector Service Port issue in YarnConfiguration. Contributed by Prabha Manepalli.
HDFS-13746. Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh (Contributed by Siyao Meng via Daniel Templeton)
YARN-8318. [UI2] IP address in component page shows N/A. Contributed by Yesha Vora.
YARN-8474. Fixed ApiServiceClient kerberos negotiation. Contributed by Billie Rinaldi
YARN-8667. Cleanup symlinks when container restarted by NM.
YARN-8656. container-executor should not write cgroup tasks files for docker containers. Contributed by Jim Brennan
YARN-8614. Fix few annotation typos in YarnConfiguration. Contributed by Sen Zhao.
HDFS-13788. Update EC documentation about rack fault tolerance. Contributed by Kitti Nanasi.
HDFS-13758. DatanodeManager should throw exception if it has BlockRecoveryCommand but the block is not under construction. Contributed by chencan.
YARN-8640. Restore previous state in container-executor after failure. Contributed by Jim Brennan
HDFS-13738. fsck -list-corruptfileblocks has infinite loop if user is not privileged. Contributed by Yuen-Kuei Hsueh.
HDFS-13813. Exit NameNode if dangling child inode is detected when saving FsImage. Contributed by Siyao Meng.
HDFS-13786. EC: Display erasure coding policy for sub-directories is not working. Contributed by Ayush Saxena.
HDFS-13735. Make QJM HTTP URL connection timeout configurable. Contributed by Chao Sun.
HDFS-13721. NPE in DataNode due to uninitialized DiskBalancer.
HDFS-13823. NameNode UI : "Utilities -> Browse the file system -> open a file -> Head the file" is not working. Contributed by Nanda kumar.
HADOOP-15638. KMS Accept Queue Size default changed from 500 to 128 in Hadoop 3.x. Contributed by Wei-Chiu Chuang.
Remove super user privilege requirement for NameNode.getServiceStatus. Contributed by Chao Sun.
YARN-8520. Document best practice for user management. Contributed by Eric Yang
HDFS-13721. NPE in DataNode due to uninitialized DiskBalancer.
YARN-8575. Avoid committing allocation proposal to unavailable nodes in async scheduling. Contributed by Tao Yang.
YARN-8521. NPE in AllocationTagsManager when a container is removed more than once. Contributed by Weiwei Yang.
YARN-8136. Add version attribute to site doc examples and quickstart. (Eric Yang via wangda)
YARN-8588. Logging improvements for better debuggability. (Suma Shivaprasad via wangda)
YARN-8559. Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint. Contributed by Weiwei Yang.
YARN-8331. Race condition in NM container launched after done. Contributed by Pradeep Ambati
YARN-8633. Update DataTables version in yarn-common in line with JQuery 3 upgrade. Contributed by Akhil PB.
YARN-8568. Replace the deprecated zk-address property in the HA config example in ResourceManagerHA.md (bsteinbach via rkanter)
YARN-8601. Print ExecutionType in Container report CLI. Contributed by Bilwa S T.
HDFS-13728. Disk Balancer should not fail if volume usage is greater than capacity. Contributed by Stephen O'Donnell.
HADOOP-15547. WASB: improve listStatus performance. Contributed by Thomas Marquardt.
YARN-8407. Container launch exception in AM log should be printed in ERROR level. (Yesha Vora via wangda)
YARN-8407. Container launch exception in AM log should be printed in ERROR level. (Yesha Vora via wangda)
YARN-8629. Container cleanup fails while trying to delete Cgroups. (Suma Shivaprasad via wangda)
YARN-8624. Updated verbiage around entry point support. Contributed by Craig Condit
YARN-8287. Update documentation and yarn-default related to the Docker runtime. Contributed by Craig Condit
YARN-8615. [UI2] Resource Usage tab shows only memory related info. No info available for vcores/gpu. Contributed by Akhil PB.
YARN-8620. [UI2] YARN Services UI new submission failures are not debuggable. Contributed by Akhil PB.
YARN-8608. [UI2] No information available per application appAttempt about 'Total Outstanding Resource Requests'. Contributed by Akhil PB.
YARN-8603. [UI2] Latest run application should be listed first in the RM UI. Contributed by Akhil PB.
YARN-8263. DockerClient still touches hadoop.tmp.dir. Contributed by Craig Condit
YARN-8592. [UI2] rmip:port/ui2 endpoint shows a blank page in windows OS and Chrome browser. Contributed by Akhil PB.
YARN-8594. [UI2] Display current logged in user. Contributed by Akhil PB.
YARN-8593. Add RM web service endpoint to get user information. Contributed by Akhil PB.
YARN-8610. Fixed initiate upgrade error message. Contributed by Chandni Singh
YARN-8403. Change the log level for fail to download resource from INFO to ERROR. Contributed by Eric Yang
YARN-8595. [UI2] Container diagnostic information is missing from container page. Contributed by Akhil PB.
YARN-8318. [UI2] IP address in component page shows N/A. Contributed by Yesha Vora.
YARN-8403. Change the log level for fail to download resource from INFO to ERROR. Contributed by Eric Yang
YARN-8397. Potential thread leak in ActivitiesManager. Contributed by Rohith Sharma K S.
YARN-8579. Recover NMToken of previous attempted component data. Contributed by Gour Saha
YARN-7974. Allow updating application tracking url after registration. Contributed by Jonathan Hung
HADOOP-15637. LocalFs#listLocatedStatus does not filter out hidden .crc files. Contributed by Erik Krogen.
YARN-8584. Several typos in Log Aggregation related classes. Contributed by Szilard Nemeth.
HDFS-12716. 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available. Contributed by Ranith Sardar and usharani
HADOOP-15607. AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream. Contributed by Jinhu Wu.
HDFS-13727. Log full stack trace if DiskBalancer exits with an unhandled exception. Contributed by Gabor Bota.
BUG-10842: 5HADOOP-15609. Retry KMS calls when SSLHandshakeException occurs. Contributed by Kitti Nanasi.
YARN-8360. Improve YARN service restart policy and node manager auto restart policy. Contributed by Suma Shivaprasad
YARN-8571. Validate service principal format prior to launching yarn service. Contributed by Eric Yang
YARN-8429. Improve diagnostic message when artifact is not set properly. Contributed by Gour Saha
YARN-8361. Change App Name Placement Rule to use App Name instead of App Id for configuration. Contributed by Zian Chen.
HADOOP-15382. Log kinit output in credential renewal thread. Contributed by Gabor Bota.
YARN-8016. Refine PlacementRule interface and add a app-name queue mapping rule as an example. Contributed by Zian Chen.
HADOOP-15560. ABFS: removed dependency injection and unnecessary dependencies. Contributed by Da Zhou.
HADOOP-15407. HADOOP-15540. Support Windows Azure Storage - Blob file system "ABFS" in Hadoop: Core Commit.
BUG-107179. YARN-8512. ATSv2 entities are not published to HBase from second attempt onwards. Contributed by Rohith Sharma K S.
BUG-106712. YARN container-executor security issue
BUG-106712. Disable mounting cgroups by default
BUG-103209. HADOOP-15483. Additional changes in jquery.jstree.js.
BUG-106392. [YARN-8474] sleeper service fails to launch with Authentication Required in upgrade cluster
BUG-96730. [YARN-8108] RM metrics api throws GSS exception in kerberized environment
YARN-8426
BUG-101524. YARN webappcontext for UI2 should inherit all filters from default context.
BUG-103209:Dependency upgrades related to blackduck scan.
HDFS-12978. Fine-grained locking while consuming journal stream. Contributed by Konstantin Shvachko. (cherry picked from commit 442dd87dcd3e9913a1d97793fde879ce51d701e7)
BUG-104243:Dependency jackson-databind dependency related to blackduck scan.
HADOOP-15299. Bump Jackson 2 version to Jackson 2.9.x.
BUG-103055 Remove aiyun, openstack from hadoop-cloud-project
BUG-102169 Include Google Connector in Hadoop
BUG-102156. HDFS-13512. WebHdfs getFileStatus doesn't return ecPolicy. Contributed by Ajay Kumar.
BUG-101422. Disable HDFS and Hadoop unit tests that are flaky in Mawo.
BUG-65303. NameNode, DataNode, SecondaryNameNode should shutdown if out of memory error occurs. Extension of BUG-5700 contributed by Chris Nauroth.
BUG-100895. Ignore UTs failing due to environmental issues.
BUG-100252. HDFS-13433. webhdfs requests can be routed incorrectly in federated cluster. Contributed by Arpit Agarwal.
BUG-121054. HADOOP-15281. Distcp to add no-rename copy option.