`

升级hadoop0.20.2到hadoop-0.21.0

HadoopMapreduceHBaseApachemaven
阅读更多

按照新的文档来 更新配置: http://hadoop.apache.org/common/docs/current/cluster_setup.html

 

发现多了很多东西,新的文档也比以前的详细,好的多.

 

不过此次只为了升级hadoop0.20.2到hadoop-0.21.0

 

 

看看发布的版本:http://hadoop.apache.org/common/releases.html

整整半年了,等的好辛苦,等得Hbase也很辛苦。 本次版本的更新，修复了N多的BUG ，看看吧：

http://hadoop.apache.org/common/docs/r0.21.0/changes.html

着实吓了一跳：

  • INCOMPATIBLE CHANGES    (31)
    1. HADOOP-4895 . Remove deprecated methods DFSClient.getHints(..) and DFSClient.isDirectory(..).
      (szetszwo)
    2. HADOOP-4941 . Remove deprecated FileSystem methods: getBlockSize(Path f), getLength(Path f) and getReplication(Path src).
      (szetszwo)
    3. HADOOP-4648 . Remove obsolete, deprecated InMemoryFileSystem and ChecksumDistributedFileSystem.
      (cdouglas via szetszwo)
    4. HADOOP-4940 . Remove a deprecated method FileSystem.delete(Path f).
      (Enis Soztutar via szetszwo)
    5. HADOOP-4010 . Change semantics for LineRecordReader to read an additional line per split- rather than moving back one character in the stream- to work with splittable compression codecs.
      (Abdul Qadeer via cdouglas)
    6. HADOOP-5094 . Show hostname and separate live/dead datanodes in DFSAdmin report.
      (Jakob Homan via szetszwo)
    7. HADOOP-4942 . Remove deprecated FileSystem methods getName() and getNamed(String name, Configuration conf).
      (Jakob Homan via szetszwo)
    8. HADOOP-5486 . Removes the CLASSPATH string from the command line and instead exports it in the environment.
      (Amareshwari Sriramadasu via ddas)
    9. HADOOP-2827 . Remove deprecated NetUtils::getServerAddress.
      (cdouglas)
    10. HADOOP-5681 . Change examples RandomWriter and RandomTextWriter to use new mapreduce API.
      (Amareshwari Sriramadasu via sharad)
    11. HADOOP-5680 . Change org.apache.hadoop.examples.SleepJob to use new mapreduce api.
      (Amareshwari Sriramadasu via sharad)
    12. HADOOP-5699 . Change org.apache.hadoop.examples.PiEstimator to use new mapreduce api.
      (Amareshwari Sriramadasu via sharad)
    13. HADOOP-5720 . Introduces new task types - JOB_SETUP, JOB_CLEANUP and TASK_CLEANUP. Removes the isMap methods from TaskID/TaskAttemptID classes.
      (ddas)
    14. HADOOP-5668 . Change TotalOrderPartitioner to use new API.
      (Amareshwari Sriramadasu via cdouglas)
    15. HADOOP-5738 . Split "waiting_tasks" JobTracker metric into waiting maps and waiting reduces.
      (Sreekanth Ramakrishnan via cdouglas)
    16. HADOOP-5679 . Resolve findbugs warnings in core/streaming/pipes/examples.
      (Jothi Padmanabhan via sharad)
    17. HADOOP-4359 . Support for data access authorization checking on Datanodes.
      (Kan Zhang via rangadi)
    18. HADOOP-5690 . Change org.apache.hadoop.examples.DBCountPageView to use new mapreduce api.
      (Amareshwari Sriramadasu via sharad)
    19. HADOOP-5694 . Change org.apache.hadoop.examples.dancing to use new mapreduce api.
      (Amareshwari Sriramadasu via sharad)
    20. HADOOP-5696 . Change org.apache.hadoop.examples.Sort to use new mapreduce api.
      (Amareshwari Sriramadasu via sharad)
    21. HADOOP-5698 . Change org.apache.hadoop.examples.MultiFileWordCount to use new mapreduce api.
      (Amareshwari Sriramadasu via sharad)
    22. HADOOP-5913 . Provide ability to an administrator to stop and start job queues.
      (Rahul Kumar Singh and Hemanth Yamijala via yhemanth)
    23. MAPREDUCE-711. Removed Distributed Cache from Common, to move it under Map/Reduce.
      (Vinod Kumar Vavilapalli via yhemanth)
    24. HADOOP-6201 . Change FileSystem::listStatus contract to throw FileNotFoundException if the directory does not exist, rather than letting this be implementation-specific.
      (Jakob Homan via cdouglas)
    25. HADOOP-6230 . Moved process tree and memory calculator related classes from Common to Map/Reduce.
      (Vinod Kumar Vavilapalli via yhemanth)
    26. HADOOP-6203 . FsShell rm/rmr error message indicates exceeding Trash quota and suggests using -skpTrash, when moving to trash fails.
      (Boris Shkolnik via suresh)
    27. HADOOP-6303 . Eclipse .classpath template has outdated jar files and is missing some new ones.
      (cos)
    28. HADOOP-6396 . Fix uninformative exception message when unable to parse umask.
      (jghoman)
    29. HADOOP-6299 . Reimplement the UserGroupInformation to use the OS specific and Kerberos JAAS login.
      (omalley)
    30. HADOOP-6686 . Remove redundant exception class name from the exception message for the exceptions thrown at RPC client.
      (suresh)
    31. HADOOP-6701 . Fix incorrect exit codes returned from chmod, chown and chgrp commands from FsShell.
      (Ravi Phulari via suresh)
  • NEW FEATURES    (59)
    1. HADOOP-6332 . Large-scale Automated Test Framework.
      (sharad, Sreekanth Ramakrishnan, at all via cos)
    2. HADOOP-4268 . Change fsck to use ClientProtocol methods so that the corresponding permission requirement for running the ClientProtocol methods will be enforced.
      (szetszwo)
    3. HADOOP-3953 . Implement sticky bit for directories in HDFS.
      (Jakob Homan via szetszwo)
    4. HADOOP-4368 . Implement df in FsShell to show the status of a FileSystem.
      (Craig Macdonald via szetszwo)
    5. HADOOP-3741 . Add a web ui to the SecondaryNameNode for showing its status.
      (szetszwo)
    6. HADOOP-5018 . Add pipelined writers to Chukwa.
      (Ari Rabkin via cdouglas)
    7. HADOOP-5052 . Add an example computing exact digits of pi using the Bailey-Borwein-Plouffe algorithm. (Tsz Wo (Nicholas), SZE via cdouglas)
    8. HADOOP-4927 . Adds a generic wrapper around outputformat to allow creation of output on demand
      (Jothi Padmanabhan via ddas)
    9. HADOOP-5144 . Add a new DFSAdmin command for changing the setting of restore failed storage replicas in namenode.
      (Boris Shkolnik via szetszwo)
    10. HADOOP-5258 . Add a new DFSAdmin command to print a tree of the rack and datanode topology as seen by the namenode.
      (Jakob Homan via szetszwo)
    11. HADOOP-4756 . A command line tool to access JMX properties on NameNode and DataNode.
      (Boris Shkolnik via rangadi)
    12. HADOOP-4539 . Introduce backup node and checkpoint node.
      (shv)
    13. HADOOP-5363 . Add support for proxying connections to multiple clusters with different versions to hdfsproxy.
      (Zhiyong Zhang via cdouglas)
    14. HADOOP-5528 . Add a configurable hash partitioner operating on ranges of BinaryComparable keys.
      (Klaas Bosteels via shv)
    15. HADOOP-5257 . HDFS servers may start and stop external components through a plugin interface.
      (Carlos Valiente via dhruba)
    16. HADOOP-5450 . Add application-specific data types to streaming's typed bytes interface.
      (Klaas Bosteels via omalley)
    17. HADOOP-5518 . Add contrib/mrunit, a MapReduce unit test framework.
      (Aaron Kimball via cutting)
    18. HADOOP-5469 . Add /metrics servlet to daemons, providing metrics over HTTP as either text or JSON.
      (Philip Zeyliger via cutting)
    19. HADOOP-5467 . Introduce offline fsimage image viewer.
      (Jakob Homan via shv)
    20. HADOOP-5752 . Add a new hdfs image processor, Delimited, to oiv.
      (Jakob Homan via szetszwo)
    21. HADOOP-5266 . Adds the capability to do mark/reset of the reduce values iterator in the Context object API.
      (Jothi Padmanabhan via ddas)
    22. HADOOP-5745 . Allow setting the default value of maxRunningJobs for all pools.
      (dhruba via matei)
    23. HADOOP-5643 . Adds a way to decommission TaskTrackers while the JobTracker is running.
      (Amar Kamat via ddas)
    24. HADOOP-4829 . Allow FileSystem shutdown hook to be disabled.
      (Todd Lipcon via tomwhite)
    25. HADOOP-5815 . Sqoop: A database import tool for Hadoop.
      (Aaron Kimball via tomwhite)
    26. HADOOP-4861 . Add disk usage with human-readable size (-duh).
      (Todd Lipcon via tomwhite)
    27. HADOOP-5844 . Use mysqldump when connecting to local mysql instance in Sqoop.
      (Aaron Kimball via tomwhite)
    28. HADOOP-5976 . Add a new command, classpath, to the hadoop script.
      (Owen O'Malley and Gary Murry via szetszwo)
    29. HADOOP-6120 . Add support for Avro specific and reflect data.
      (sharad via cutting)
    30. HADOOP-6226 . Moves BoundedByteArrayOutputStream from the tfile package to the io package and makes it available to other users (MAPREDUCE-318).
      (Jothi Padmanabhan via ddas)
    31. HADOOP-6105 . Adds support for automatically handling deprecation of configuration keys.
      (V.V.Chaitanya Krishna via yhemanth)
    32. HADOOP-6235 . Adds new method to FileSystem for clients to get server defaults.
      (Kan Zhang via suresh)
    33. HADOOP-6234 . Add new option dfs.umaskmode to set umask in configuration to use octal or symbolic instead of decimal.
      (Jakob Homan via suresh)
    34. HADOOP-5073 . Add annotation mechanism for interface classification.
      (Jakob Homan via suresh)
    35. HADOOP-4012 . Provide splitting support for bzip2 compressed files.
      (Abdul Qadeer via cdouglas)
    36. HADOOP-6246 . Add backward compatibility support to use deprecated decimal umask from old configuration.
      (Jakob Homan via suresh)
    37. HADOOP-4952 . Add new improved file system interface FileContext for the application writer
      (Sanjay Radia via suresh)
    38. HADOOP-6170 . Add facility to tunnel Avro RPCs through Hadoop RPCs. This permits one to take advantage of both Avro's RPC versioning features and Hadoop's proven RPC scalability.
      (cutting)
    39. HADOOP-6267 . Permit building contrib modules located in external source trees.
      (Todd Lipcon via cutting)
    40. HADOOP-6240 . Add new FileContext rename operation that posix compliant that allows overwriting existing destination.
      (suresh)
    41. HADOOP-6204 . Implementing aspects development and fault injeciton framework for Hadoop
      (cos)
    42. HADOOP-6313 . Implement Syncable interface in FSDataOutputStream to expose flush APIs to application users.
      (Hairong Kuang via suresh)
    43. HADOOP-6284 . Add a new parameter, HADOOP_JAVA_PLATFORM_OPTS, to hadoop-config.sh so that it allows setting java command options for JAVA_PLATFORM.
      (Koji Noguchi via szetszwo)
    44. HADOOP-6337 . Updates FilterInitializer class to be more visible, and the init of the class is made to take a Configuration argument.
      (Jakob Homan via ddas)
    45. Hadoop-6223. Add new file system interface AbstractFileSystem with implementation of some file systems that delegate to old FileSystem.
      (Sanjay Radia via suresh)
    46. HADOOP-6433 . Introduce asychronous deletion of files via a pool of threads. This can be used to delete files in the Distributed Cache.
      (Zheng Shao via dhruba)
    47. HADOOP-6415 . Adds a common token interface for both job token and delegation token.
      (Kan Zhang via ddas)
    48. HADOOP-6408 . Add a /conf servlet to dump running configuration.
      (Todd Lipcon via tomwhite)
    49. HADOOP-6520 . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable.
      (ddas)
    50. HADOOP-6419 . Adds SASL based authentication to RPC.
      (Kan Zhang via ddas)
    51. HADOOP-6510 . Adds a way for superusers to impersonate other users in a secure environment.
      (Jitendra Nath Pandey via ddas)
    52. HADOOP-6421 . Adds Symbolic links to FileContext, AbstractFileSystem. It also adds a limited implementation for the local file system (RawLocalFs) that allows local symlinks.
      (Eli Collins via Sanjay Radia)
    53. HADOOP-6577 . Add hidden configuration option "ipc.server.max.response.size" to change the default 1 MB, the maximum size when large IPC handler response buffer is reset.
      (suresh)
    54. HADOOP-6568 . Adds authorization for the default servlets.
      (Vinod Kumar Vavilapalli via ddas)
    55. HADOOP-6586 . Log authentication and authorization failures and successes for RPC
      (boryas)
    56. HADOOP-6580 . UGI should contain authentication method.
      (jnp via boryas)
    57. HADOOP-6657 . Add a capitalization method to StringUtils for MAPREDUCE-1545.
      (Luke Lu via Steve Loughran)
    58. HADOOP-6692 . Add FileContext#listStatus that returns an iterator.
      (hairong)
    59. HADOOP-6869 . Functionality to create file or folder on a remote daemon side
      (Vinay Thota via cos)
  • IMPROVEMENTS    (198)
    1. HADOOP-6798 . Align Ivy version for all Hadoop subprojects.
      (cos)
    2. HADOOP-6777 . Implement a functionality for suspend and resume a process.
      (Vinay Thota via cos)
    3. HADOOP-6772 . Utilities for system tests specific.
      (Vinay Thota via cos)
    4. HADOOP-6771 . Herriot's artifact id for Maven deployment should be set to hadoop-core-instrumented
      (cos)
    5. HADOOP-6752 . Remote cluster control functionality needs JavaDocs improvement (Balaji Rajagopalan via cos).
    6. HADOOP-4565 . Added CombineFileInputFormat to use data locality information to create splits.
      (dhruba via zshao)
    7. HADOOP-4936 . Improvements to TestSafeMode.
      (shv)
    8. HADOOP-4985 . Remove unnecessary "throw IOException" declarations in FSDirectory related methods.
      (szetszwo)
    9. HADOOP-5017 . Change NameNode.namesystem declaration to private.
      (szetszwo)
    10. HADOOP-4794 . Add branch information from the source version control into the version information that is compiled into Hadoop.
      (cdouglas via omalley)
    11. HADOOP-5070 . Increment copyright year to 2009, remove assertions of ASF copyright to licensed files. (Tsz Wo (Nicholas), SZE via cdouglas)
    12. HADOOP-5037 . Deprecate static FSNamesystem.getFSNamesystem().
      (szetszwo)
    13. HADOOP-5088 . Include releaseaudit target as part of developer test-patch target.
      (Giridharan Kesavan via nigel)
    14. HADOOP-2721 . Uses setsid when creating new tasks so that subprocesses of this process will be within this new session (and this process will be the process leader for all the subprocesses). Killing the process leader, or the main Java task in Hadoop's case, kills the entire subtree of processes.
      (Ravi Gummadi via ddas)
    15. HADOOP-5097 . Remove static variable JspHelper.fsn, a static reference to a non-singleton FSNamesystem object.
      (szetszwo)
    16. HADOOP-3327 . Improves handling of READ_TIMEOUT during map output copying.
      (Amareshwari Sriramadasu via ddas)
    17. HADOOP-5124 . Choose datanodes randomly instead of starting from the first datanode for providing fairness.
      (hairong via szetszwo)
    18. HADOOP-4930 . Implement a Linux native executable that can be used to launch tasks as users.
      (Sreekanth Ramakrishnan via yhemanth)
    19. HADOOP-5122 . Fix format of fs.default.name value in libhdfs test conf.
      (Craig Macdonald via tomwhite)
    20. HADOOP-5038 . Direct daemon trace to debug log instead of stdout.
      (Jerome Boulon via cdouglas)
    21. HADOOP-5101 . Improve packaging by adding 'all-jars' target building core, tools, and example jars. Let findbugs depend on this rather than the 'tar' target.
      (Giridharan Kesavan via cdouglas)
    22. HADOOP-4868 . Splits the hadoop script into three parts - bin/hadoop, bin/mapred and bin/hdfs.
      (Sharad Agarwal via ddas)
    23. HADOOP-1722 . Adds support for TypedBytes and RawBytes in Streaming.
      (Klaas Bosteels via ddas)
    24. HADOOP-4220 . Changes the JobTracker restart tests so that they take much less time.
      (Amar Kamat via ddas)
    25. HADOOP-4885 . Try to restore failed name-node storage directories at checkpoint time.
      (Boris Shkolnik via shv)
    26. HADOOP-5209 . Update year to 2009 for javadoc.
      (szetszwo)
    27. HADOOP-5279 . Remove unnecessary targets from test-patch.sh.
      (Giridharan Kesavan via nigel)
    28. HADOOP-5120 . Remove the use of FSNamesystem.getFSNamesystem() from UpgradeManagerNamenode and UpgradeObjectNamenode.
      (szetszwo)
    29. HADOOP-5222 . Add offset to datanode clienttrace.
      (Lei Xu via cdouglas)
    30. HADOOP-5240 . Skip re-building javadoc when it is already up-to-date.
      (Aaron Kimball via cutting)
    31. HADOOP-5042 . Add a cleanup stage to log rollover in Chukwa appender.
      (Jerome Boulon via cdouglas)
    32. HADOOP-5264 . Removes redundant configuration object from the TaskTracker.
      (Sharad Agarwal via ddas)
    33. HADOOP-5232 . Enable patch testing to occur on more than one host.
      (Giri Kesavan via nigel)
    34. HADOOP-4546 . Fix DF reporting for AIX.
      (Bill Habermaas via cdouglas)
    35. HADOOP-5023 . Add Tomcat support to HdfsProxy.
      (Zhiyong Zhang via cdouglas)
    36. HADOOP-5317 . Provide documentation for LazyOutput Feature.
      (Jothi Padmanabhan via johan)
    37. HADOOP-5455 . Document rpc metrics context to the extent dfs, mapred, and jvm contexts are documented.
      (Philip Zeyliger via cdouglas)
    38. HADOOP-5358 . Provide scripting functionality to the synthetic load generator.
      (Jakob Homan via hairong)
    39. HADOOP-5442 . Paginate jobhistory display and added some search capabilities.
      (Amar Kamat via acmurthy)
    40. HADOOP-4842 . Streaming now allows specifiying a command for the combiner.
      (Amareshwari Sriramadasu via ddas)
    41. HADOOP-5196 . avoiding unnecessary byte[] allocation in SequenceFile.CompressedBytes and SequenceFile.UncompressedBytes.
      (hong tang via mahadev)
    42. HADOOP-4655 . New method FileSystem.newInstance() that always returns a newly allocated FileSystem object.
      (dhruba)
    43. HADOOP-4788 . Set Fair scheduler to assign both a map and a reduce on each heartbeat by default.
      (matei)
    44. HADOOP-5491 . In contrib/index, better control memory usage.
      (Ning Li via cutting)
    45. HADOOP-5423 . Include option of preserving file metadata in SequenceFile::sort.
      (Michael Tamm via cdouglas)
    46. HADOOP-5331 . Add support for KFS appends.
      (Sriram Rao via cdouglas)
    47. HADOOP-4365 . Make Configuration::getProps protected in support of meaningful subclassing.
      (Steve Loughran via cdouglas)
    48. HADOOP-2413 . Remove the static variable FSNamesystem.fsNamesystemObject.
      (Konstantin Shvachko via szetszwo)
    49. HADOOP-4584 . Improve datanode block reports and associated file system scan to avoid interefering with normal datanode operations.
      (Suresh Srinivas via rangadi)
    50. HADOOP-5502 . Documentation for backup and checkpoint nodes.
      (Jakob Homan via shv)
    51. HADOOP-5485 . Mask actions in the fair scheduler's servlet UI based on value of webinterface.private.actions.
      (Vinod Kumar Vavilapalli via yhemanth)
    52. HADOOP-5581 . HDFS should throw FileNotFoundException when while opening a file that does not exist.
      (Brian Bockelman via rangadi)
    53. HADOOP-5509 . PendingReplicationBlocks does not start monitor in the constructor.
      (shv)
    54. HADOOP-5494 . Modify sorted map output merger to lazily read values, rather than buffering at least one record for each segment.
      (Devaraj Das via cdouglas)
    55. HADOOP-5396 . Provide ability to refresh queue ACLs in the JobTracker without having to restart the daemon.
      (Sreekanth Ramakrishnan and Vinod Kumar Vavilapalli via yhemanth)
    56. HADOOP-4490 . Provide ability to run tasks as job owners.
      (Sreekanth Ramakrishnan via yhemanth)
    57. HADOOP-5697 . Change org.apache.hadoop.examples.Grep to use new mapreduce api.
      (Amareshwari Sriramadasu via sharad)
    58. HADOOP-5625 . Add operation duration to clienttrace.
      (Lei Xu via cdouglas)
    59. HADOOP-5705 . Improve TotalOrderPartitioner efficiency by updating the trie construction.
      (Dick King via cdouglas)
    60. HADOOP-5589 . Eliminate source limit of 64 for map-side joins imposed by TupleWritable encoding.
      (Jingkei Ly via cdouglas)
    61. HADOOP-5734 . Correct block placement policy description in HDFS Design document.
      (Konstantin Boudnik via shv)
    62. HADOOP-5657 . Validate data in TestReduceFetch to improve merge test coverage.
      (cdouglas)
    63. HADOOP-5613 . Change S3Exception to checked exception.
      (Andrew Hitchcock via tomwhite)
    64. HADOOP-5717 . Create public enum class for the Framework counters in org.apache.hadoop.mapreduce.
      (Amareshwari Sriramadasu via sharad)
    65. HADOOP-5217 . Split AllTestDriver for core, hdfs and mapred.
      (sharad)
    66. HADOOP-5364 . Add certificate expiration warning to HsftpFileSystem and HDFS proxy.
      (Zhiyong Zhang via cdouglas)
    67. HADOOP-5733 . Add map/reduce slot capacity and blacklisted capacity to JobTracker metrics.
      (Sreekanth Ramakrishnan via cdouglas)
    68. HADOOP-5596 . Add EnumSetWritable.
      (He Yongqiang via szetszwo)
    69. HADOOP-5727 . Simplify hashcode for ID types.
      (Shevek via cdouglas)
    70. HADOOP-5500 . In DBOutputFormat, where field names are absent permit the number of fields to be sufficient to construct the select query.
      (Enis Soztutar via cdouglas)
    71. HADOOP-5081 . Split TestCLI into HDFS, Mapred and Core tests.
      (sharad)
    72. HADOOP-5015 . Separate block management code from FSNamesystem.
      (Suresh Srinivas via szetszwo)
    73. HADOOP-5080 . Add new test cases to TestMRCLI and TestHDFSCLI
      (V.Karthikeyan via nigel)
    74. HADOOP-5135 . Splits the tests into different directories based on the package. Four new test targets have been defined - run-test-core, run-test-mapred, run-test-hdfs and run-test-hdfs-with-mr.
      (Sharad Agarwal via ddas)
    75. HADOOP-5771 . Implements unit tests for LinuxTaskController.
      (Sreekanth Ramakrishnan and Vinod Kumar Vavilapalli via yhemanth)
    76. HADOOP-5419 . Provide a facility to query the Queue ACLs for the current user.
      (Rahul Kumar Singh via yhemanth)
    77. HADOOP-5780 . Improve per block message prited by "-metaSave" in HDFS.
      (Raghu Angadi)
    78. HADOOP-5823 . Added a new class DeprecatedUTF8 to help with removing UTF8 related javac warnings. These warnings are removed in FSEditLog.java as a use case.
      (Raghu Angadi)
    79. HADOOP-5824 . Deprecate DataTransferProtocol.OP_READ_METADATA and remove the corresponding unused codes.
      (Kan Zhang via szetszwo)
    80. HADOOP-5721 . Factor out EditLogFileInputStream and EditLogFileOutputStream into independent classes.
      (Luca Telloli & Flavio Junqueira via shv)
    81. HADOOP-5838 . Fix a few javac warnings in HDFS.
      (Raghu Angadi)
    82. HADOOP-5854 . Fix a few "Inconsistent Synchronization" warnings in HDFS.
      (Raghu Angadi)
    83. HADOOP-5369 . Small tweaks to reduce MapFile index size.
      (Ben Maurer via sharad)
    84. HADOOP-5858 . Eliminate UTF8 and fix warnings in test/hdfs-with-mr package.
      (shv)
    85. HADOOP-5866 . Move DeprecatedUTF8 from o.a.h.io to o.a.h.hdfs since it may not be used outside hdfs.
      (Raghu Angadi)
    86. HADOOP-5857 . Move normal java methods from hdfs .jsp files to .java files.
      (szetszwo)
    87. HADOOP-5873 . Remove deprecated methods randomDataNode() and getDatanodeByIndex(..) in FSNamesystem.
      (szetszwo)
    88. HADOOP-5572 . Improves the progress reporting for the sort phase for both maps and reduces.
      (Ravi Gummadi via ddas)
    89. HADOOP-5839 . Fix EC2 scripts to allow remote job submission.
      (Joydeep Sen Sarma via tomwhite)
    90. HADOOP-5877 . Fix javac warnings in TestHDFSServerPorts, TestCheckpoint, TestNameEditsConfig, TestStartup and TestStorageRestore.
      (Jakob Homan via shv)
    91. HADOOP-5438 . Provide a single FileSystem method to create or open-for-append to a file.
      (He Yongqiang via dhruba)
    92. HADOOP-5472 . Change DistCp to support globbing of input paths.
      (Dhruba Borthakur and Rodrigo Schmidt via szetszwo)
    93. HADOOP-5175 . Don't unpack libjars on classpath.
      (Todd Lipcon via tomwhite)
    94. HADOOP-5620 . Add an option to DistCp for preserving modification and access times.
      (Rodrigo Schmidt via szetszwo)
    95. HADOOP-5664 . Change map serialization so a lock is obtained only where contention is possible, rather than for each write.
      (cdouglas)
    96. HADOOP-5896 . Remove the dependency of GenericOptionsParser on Option.withArgPattern.
      (Giridharan Kesavan and Sharad Agarwal via sharad)
    97. HADOOP-5784 . Makes the number of heartbeats that should arrive a second at the JobTracker configurable.
      (Amareshwari Sriramadasu via ddas)
    98. HADOOP-5955 . Changes TestFileOuputFormat so that is uses LOCAL_MR instead of CLUSTER_MR.
      (Jothi Padmanabhan via das)
    99. HADOOP-5948 . Changes TestJavaSerialization to use LocalJobRunner instead of MiniMR/DFS cluster.
      (Jothi Padmanabhan via das)
    100. HADOOP-2838 . Add mapred.child.env to pass environment variables to tasktracker's child processes.
      (Amar Kamat via sharad)
    101. HADOOP-5961 . DataNode process understand generic hadoop command line options (like -Ddfs.property=value).
      (Raghu Angadi)
    102. HADOOP-5938 . Change org.apache.hadoop.mapred.jobcontrol to use new api.
      (Amareshwari Sriramadasu via sharad)
    103. HADOOP-2141 . Improves the speculative execution heuristic. The heuristic is currently based on the progress-rates of tasks and the expected time to complete. Also, statistics about trackers are collected, and speculative tasks are not given to the ones deduced to be slow.
      (Andy Konwinski and ddas)
    104. HADOOP-5952 . Change "-1 tests included" wording in test-patch.sh.
      (Gary Murry via szetszwo)
    105. HADOOP-6106 . Provides an option in ShellCommandExecutor to timeout commands that do not complete within a certain amount of time.
      (Sreekanth Ramakrishnan via yhemanth)
    106. HADOOP-5925 . EC2 scripts should exit on error.
      (tomwhite)
    107. HADOOP-6109 . Change Text to grow its internal buffer exponentially, rather than the max of the current length and the proposed length to improve performance reading large values.
      (thushara wijeratna via cdouglas)
    108. HADOOP-2366 . Support trimmed strings in Configuration.
      (Michele Catasta via szetszwo)
    109. HADOOP-6099 . The RPC module can be configured to not send period pings. The default behaviour of sending periodic pings remain unchanged.
      (dhruba)
    110. HADOOP-6142 . Update documentation and use of harchives for relative paths added in MAPREDUCE-739.
      (Mahadev Konar via cdouglas)
    111. HADOOP-6148 . Implement a fast, pure Java CRC32 calculator which outperforms java.util.zip.CRC32.
      (Todd Lipcon and Scott Carey via szetszwo)
    112. HADOOP-6146 . Upgrade to JetS3t version 0.7.1.
      (tomwhite)
    113. HADOOP-6161 . Add get/setEnum methods to Configuration.
      (cdouglas)
    114. HADOOP-6160 . Fix releaseaudit target to run on specific directories.
      (gkesavan)
    115. HADOOP-6169 . Removing deprecated method calls in TFile.
      (hong tang via mahadev)
    116. HADOOP-6176 . Add a couple package private methods to AccessTokenHandler for testing.
      (Kan Zhang via szetszwo)
    117. HADOOP-6182 . Fix ReleaseAudit warnings
      (Giridharan Kesavan and Lee Tucker via gkesavan)
    118. HADOOP-6173 . Change src/native/packageNativeHadoop.sh to package all native library files.
      (Hong Tang via szetszwo)
    119. HADOOP-6184 . Provide an API to dump Configuration in a JSON format.
      (V.V.Chaitanya Krishna via yhemanth)
    120. HADOOP-6224 . Add a method to WritableUtils performing a bounded read of an encoded String.
      (Jothi Padmanabhan via cdouglas)
    121. HADOOP-6133 . Add a caching layer to Configuration::getClassByName to alleviate a performance regression introduced in a compatibility layer.
      (Todd Lipcon via cdouglas)
    122. HADOOP-6252 . Provide a method to determine if a deprecated key is set in config file.
      (Jakob Homan via suresh)
    123. HADOOP-5879 . Read compression level and strategy from Configuration for gzip compression.
      (He Yongqiang via cdouglas)
    124. HADOOP-6216 . Support comments in host files.
      (Ravi Phulari and Dmytro Molkov via szetszwo)
    125. HADOOP-6217 . Update documentation for project split.
      (Corinne Chandel via omalley)
    126. HADOOP-6268 . Add ivy jar to .gitignore.
      (Todd Lipcon via cdouglas)
    127. HADOOP-6270 . Support deleteOnExit in FileContext.
      (Suresh Srinivas via szetszwo)
    128. HADOOP-6233 . Rename configuration keys towards API standardization and backward compatibility.
      (Jithendra Pandey via suresh)
    129. HADOOP-6260 . Add additional unit tests for FileContext util methods. (Gary Murry via suresh).
    130. HADOOP-6309 . Change build.xml to run tests with java asserts.
      (Eli Collins via szetszwo)
    131. HADOOP-6326 . Hundson runs should check for AspectJ warnings and report failure if any is present
      (cos)
    132. HADOOP-6329 . Add build-fi directory to the ignore lists.
      (szetszwo)
    133. HADOOP-5107 . Use Maven ant tasks to publish the subproject jars.
      (Giridharan Kesavan via omalley)
    134. HADOOP-6343 . Log unexpected throwable object caught in RPC.
      (Jitendra Nath Pandey via szetszwo)
    135. HADOOP-6367 . Removes Access Token implementation from common.
      (Kan Zhang via ddas)
    136. HADOOP-6395 . Upgrade some libraries to be consistent across common, hdfs, and mapreduce.
      (omalley)
    137. HADOOP-6398 . Build is broken after HADOOP-6395 patch has been applied
      (cos)
    138. HADOOP-6413 . Move TestReflectionUtils to Common.
      (Todd Lipcon via tomwhite)
    139. HADOOP-6283 . Improve the exception messages thrown by FileUtil$HardLink.getLinkCount(..).
      (szetszwo)
    140. HADOOP-6279 . Add Runtime::maxMemory to JVM metrics.
      (Todd Lipcon via cdouglas)
    141. HADOOP-6305 . Unify build property names to facilitate cross-projects modifications
      (cos)
    142. HADOOP-6312 . Remove unnecessary debug logging in Configuration constructor.
      (Aaron Kimball via cdouglas)
    143. HADOOP-6366 . Reduce ivy console output to ovservable level
      (cos)
    144. HADOOP-6400 . Log errors getting Unix UGI.
      (Todd Lipcon via tomwhite)
    145. HADOOP-6346 . Add support for specifying unpack pattern regex to RunJar.unJar.
      (Todd Lipcon via tomwhite)
    146. HADOOP-6422 . Make RPC backend plugable, protocol-by-protocol, to ease evolution towards Avro.
      (cutting)
    147. HADOOP-5958 . Use JDK 1.6 File APIs in DF.java wherever possible.
      (Aaron Kimball via tomwhite)
    148. HADOOP-6222 . Core doesn't have TestCommonCLI facility.
      (cos)
    149. HADOOP-6394 . Add a helper class to simplify FileContext related tests and improve code reusability.
      (Jitendra Nath Pandey via suresh)
    150. HADOOP-4656 . Add a user to groups mapping service.
      (boryas, acmurthy)
    151. HADOOP-6435 . Make RPC.waitForProxy with timeout public.
      (Steve Loughran via tomwhite)
    152. HADOOP-6472 . add tokenCache option to GenericOptionsParser for passing file with secret keys to a map reduce job.
      (boryas)
    153. HADOOP-3205 . Read multiple chunks directly from FSInputChecker subclass into user buffers.
      (Todd Lipcon via tomwhite)
    154. HADOOP-6479 . TestUTF8 assertions could fail with better text.
      (Steve Loughran via tomwhite)
    155. HADOOP-6155 . Deprecate RecordIO anticipating Avro.
      (Tom White via cdouglas)
    156. HADOOP-6492 . Make some Avro serialization APIs public.
      (Aaron Kimball via cutting)
    157. HADOOP-6497 . Add an adapter for Avro's SeekableInput interface, so that Avro can read FileSystem data.
      (Aaron Kimball via cutting)
    158. HADOOP-6495 . Identifier should be serialized after the password is created In Token constructor
      (jnp via boryas)
    159. HADOOP-6518 . Makes the UGI honor the env var KRB5CCNAME.
      (Owen O'Malley via ddas)
    160. HADOOP-6531 . Enhance FileUtil with an API to delete all contents of a directory.
      (Amareshwari Sriramadasu via yhemanth)
    161. HADOOP-6547 . Move DelegationToken into Common, so that it can be used by MapReduce also.
      (devaraj via omalley)
    162. HADOOP-6552 . Puts renewTGT=true and useTicketCache=true for the keytab kerberos options.
      (ddas)
    163. HADOOP-6534 . Trim whitespace from directory lists initializing LocalDirAllocator.
      (Todd Lipcon via cdouglas)
    164. HADOOP-6559 . Makes the RPC client automatically re-login when the SASL connection setup fails. This is applicable only to keytab based logins.
      (Devaraj Das)
    165. HADOOP-6551 . Delegation token renewing and cancelling should provide meaningful exceptions when there are failures instead of returning false.
      (omalley)
    166. HADOOP-6583 . Captures authentication and authorization metrics.
      (ddas)
    167. HADOOP-6543 . Allows secure clients to talk to unsecure clusters.
      (Kan Zhang via ddas)
    168. HADOOP-6579 . Provide a mechanism for encoding/decoding Tokens from a url-safe string and change the commons-code library to 1.4.
      (omalley)
    169. HADOOP-6596 . Add a version field to the AbstractDelegationTokenIdentifier's serialized value.
      (omalley)
    170. HADOOP-6573 . Support for persistent delegation tokens.
      (Jitendra Pandey via shv)
    171. HADOOP-6594 . Provide a fetchdt tool via bin/hdfs.
      (jhoman via acmurthy)
    172. HADOOP-6589 . Provide better error messages when RPC authentication fails.
      (Kan Zhang via omalley)
    173. HADOOP-6599 Split existing RpcMetrics into RpcMetrics & RpcDetailedMetrics.
      (Suresh Srinivas via Sanjay Radia)
    174. HADOOP-6537 Declare more detailed exceptions in FileContext and AbstractFileSystem
      (Suresh Srinivas via Sanjay Radia)
    175. HADOOP-6486 . fix common classes to work with Avro 1.3 reflection.
      (cutting via tomwhite)
    176. HADOOP-6591 . HarFileSystem can handle paths with the whitespace characters.
      (Rodrigo Schmidt via dhruba)
    177. HADOOP-6407 . Have a way to automatically update Eclipse .classpath file when new libs are added to the classpath through Ivy.
      (tomwhite)
    178. HADOOP-3659 . Patch to allow hadoop native to compile on Mac OS X.
      (Colin Evans and Allen Wittenauer via tomwhite)
    179. HADOOP-6471 . StringBuffer -> StringBuilder - conversion of references as necessary.
      (Kay Kay via tomwhite)
    180. HADOOP-6646 . Move HarfileSystem out of Hadoop Common.
      (mahadev)
    181. HADOOP-6566 . Add methods supporting, enforcing narrower permissions on local daemon directories.
      (Arun Murthy and Luke Lu via cdouglas)
    182. HADOOP-6705 . Fix to work with 1.5 version of jiracli
      (Giridharan Kesavan)
    183. HADOOP-6658 . Exclude Private elements from generated Javadoc.
      (tomwhite)
    184. HADOOP-6635 . Install/deploy source jars to Maven repo.
      (Patrick Angeles via jghoman)
    185. HADOOP-6717 . Log levels in o.a.h.security.Groups too high
      (Todd Lipcon via jghoman)
    186. HADOOP-6667 . RPC.waitForProxy should retry through NoRouteToHostException.
      (Todd Lipcon via tomwhite)
    187. HADOOP-6677 . InterfaceAudience.LimitedPrivate should take a string not an enum.
      (tomwhite)
    188. HADOOP-678 . Remove FileContext#isFile, isDirectory, and exists.
      (Eli Collins via hairong)
    189. HADOOP-6515 . Make maximum number of http threads configurable.
      (Scott Chen via zshao)
    190. HADOOP-6563 . Add more symlink tests to cover intermediate symlinks in paths.
      (Eli Collins via suresh)
    191. HADOOP-6585 . Add FileStatus#isDirectory and isFile.
      (Eli Collins via tomwhite)
    192. HADOOP-6738 . Move cluster_setup.xml from MapReduce to Common.
      (Tom White via tomwhite)
    193. HADOOP-6794 . Move configuration and script files post split.
      (tomwhite)
    194. HADOOP-6403 . Deprecate EC2 bash scripts.
      (tomwhite)
    195. HADOOP-6769 . Add an API in FileSystem to get FileSystem instances based on users
      (ddas via boryas)
    196. HADOOP-6813 . Add a new newInstance method in FileSystem that takes a "user" as argument
      (ddas via boryas)
    197. HADOOP-6668 . Apply audience and stability annotations to classes in common.
      (tomwhite)
    198. HADOOP-6821 . Document changes to memory monitoring.
      (Hemanth Yamijala via tomwhite)
  • OPTIMIZATIONS    (12)
    1. HADOOP-5595 . NameNode does not need to run a replicator to choose a random DataNode.
      (hairong)
    2. HADOOP-5603 . Improve NameNode's block placement performance.
      (hairong)
    3. HADOOP-5638 . More improvement on block placement performance.
      (hairong)
    4. HADOOP-6180 . NameNode slowed down when many files with same filename were moved to Trash.
      (Boris Shkolnik via hairong)
    5. HADOOP-6166 . Further improve the performance of the pure-Java CRC32 implementation. (Tsz Wo (Nicholas), SZE via cdouglas)
    6. HADOOP-6271 . Add recursive and non recursive create and mkdir to FileContext.
      (Sanjay Radia via suresh)
    7. HADOOP-6261 . Add URI based tests for FileContext. (Ravi Pulari via suresh).
    8. HADOOP-6307 . Add a new SequenceFile.Reader constructor in order to support reading on un-closed file.
      (szetszwo)
    9. HADOOP-6467 . Improve the performance on HarFileSystem.listStatus(..).
      (mahadev via szetszwo)
    10. HADOOP-6569 . FsShell#cat should avoid calling unecessary getFileStatus before opening a file to read.
      (hairong)
    11. HADOOP-6689 . Add directory renaming test to existing FileContext tests.
      (Eli Collins via suresh)
    12. HADOOP-6713 . The RPC server Listener thread is a scalability bottleneck.
      (Dmytro Molkov via hairong)
  • BUG FIXES    (231)

不过，个人觉得hadoop的稳定性还是比hbase好，虽然hbase更新速度无比的快。

 

算了，还是言归正传，升级：

 

1、core-site.xml 不变

 

2、hdfs.site.xml 可以不变也可以把dfs.name.dir和dfs.data.dir更新成这样：

<property>
        <name>dfs.namenode.name.dir</name>
        <value>/data0/cloud/namenode/</value>
</description>

</property>

<property>
        <name>dfs.datanode.data.dir</name>
        <value>/data0/cloud/datanode/</value>
</property>

 不过，有个警告就是：

WARN org.apache.hadoop.hdfs.server.common.Util: Path /data0/cloud/namenode/ should be specified as a URI in configuration files. Please update hdfs configuration.
 

3、mapred-site.xml 

<!--
<property>
        <name>mapred.job.tracker</name>
        <value>name.uc.uuwatch.com:9001</value>
</property>
-->

<property>
        <name>mapreduce.jobtracker.address</name>
        <value>tracker.uc.uuwatch.com:9001</value>
</property>
 

同步所有的配置文件到到所有的节点，然后启动hadoop

 

...bin/start-all.sh

 

你会发现脚本也更新了，比之前的分类更好，控制更加细致，很好！

 

不过，可惜的是namenode和secondnamenode不能启动，分析下日志会发现需要升级文件（原来文件格式也有一定的更改，优化了吗，呵呵）：

ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.
java.io.IOException:
File system image contains an old layout version -18.
An upgrade to version -24 is required.
Please restart NameNode with -upgrade option.
 

4、OK 行了，执行名称节点的文件系统更新命令 吧

 

.../bin/hadoop namenode -upgrade

 

好，到此成功启动节点。升级完毕！

接下来，我启动hbase0.20.6试试了。

 

特别提醒：备份你的hadoop数据，不要回不到原来的版本了。请查阅 bin/hadoop namenode 相关的命令以及数据节点的备份。

简单点：可以拷贝你的原始数据到新的文件夹，更改配置，使用新文件夹的数据升级尝试，失败了也不怕。数据完好

 

hbase0.20.6是基于hadoop0.20.2的  发现升级后 RPC对不住，文件系统对不住。

更改hbade的依赖，改RPC 去叉 重新编译 部署

 

 

另外：经过修改hbase可以启动 但是不能读写表，还没有修改完毕的原因。

0
顶
2
踩
分享到：
| Solr简单的多个实例
评论
2 楼 iammonster 2010-12-17  
david.org 写道
觉得楼主还可以把文章写的更详细点。 比如0.21.0提示警告我们应当如何来避免等。

另外，楼主有测试过0.21.0么，有机会写一个在0.21.0版本遇到的问题以及解决方案如何？:)


不好意思，采用的是这个分支编译的
http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-append/
1 楼 david.org 2010-09-22  
觉得楼主还可以把文章写的更详细点。 比如0.21.0提示警告我们应当如何来避免等。

另外，楼主有测试过0.21.0么，有机会写一个在0.21.0版本遇到的问题以及解决方案如何？:)
发表评论

您还没有登录,请您登录后再发表评论

相关推荐

Magicbox
Global site tag (gtag.js) - Google Analytics