Migrating and Merging Tony Smith [email protected] Agenda

Migrating and Merging Tony Smith Tony@Perforce.Com Agenda

A Walk on the Wild Side

What we’ll cover Migrating servers between platforms Merging databases What I hope you’ll come away with A healthy respect for both! Migrating Perforce Servers Migrating

A tour of the issues From Windows to Unix/Linux From Unix/Linux to Windows Solutions past & present Migrating: Issues

Must migrate the database and the archive files. Case-sensitivity Line-ending differences Endianism Can’t simply copy db.* files over! See: TechNote 10 and TechNote 75 Migrating from Windows

Case-insensitive to case-sensitive Need to clean up use of case Case-insensitive to case-insensitive Use ‘p4d –C1’ on Unix Undocumented flag: server runs case- insensitive on Unix platforms Database is ready-to-go Just restore from checkpoint Rename all archive files to lower-case Migrating from Unix

To another Unix or Linux? No problem! To Windows? Yikes! Bad idea! Going from case-sensitive to case- insensitive can involve loss of data No easy solution Migrating: Case Sensitivity

Entities named inconsistently now appear distinct! For example: //depot/… and //Depot/… //depot/main/www/… //depot/main/WWW/… //depot/www/index.html //depot/Www/Index.html Or Client: myclient View: //depot/… //MYCLIENT/… Migrating: Solutions Past checkcase.pl Searched checkpoint for inconsistent paths p4fixcase.pl Renamed archive files to match checkpoint case Note: nothing to help you fix inconsistencies! Use ‘vi’ or wordpad on 4GB checkpoint?! Staggeringly slow (sometimes took days!) Consigned gratefully to the waste-bin of history! Migrating: The present p4migrate Written in C++ for speed Checks for many more problems Discriminates between ‘major’ and ‘minor’ inconsistencies Major: requires user to make a decision Minor: can be resolved without user involvement Uses Perforce’s mapping code to express user’s requirements Migrating: p4migrate

Works with checkpoint and archive files Pass 1 detects conflicts Producing a ‘conflict map’ User edits the conflict map Makes case-corrections (only!) Pass 2 corrects case Produces new checkpoint Can also rename archive files Designed to be restart-able Migrating: p4migrate

Version agnostic! Downloads & Docs at: http://www.perforce.com/downloads/perforce/tools/p4-migrate/ Migrating: p4migrate

TransMap

Apply Policy Translate check-point

Add to PathTree

Write Output Conflict Map

new check-point Transferring files from Windows

Filename charset translation Non-ascii characters can be translated into a different character set é = 0x82 in DOS CP 437, but 0xE9 in ISO8859-1 Filename in checkpoint is not translated! So you end up with ‘p4 verify’ saying files are ‘MISSING!’ Usually the biggest challenge! Cygwin tar known to work Network copy via Samba known not to work! Migrating: Line-ending differences

Windows uses ‘\r\n’ Unix uses ‘\n’ Either acceptable in checkpoint Must translate for all text-based archives text binary+D All stored in *,v files. Migrating: Line-ending differences

Solutions that work find . –name \*,v –print0 | \ xargs –0 perl –pi –e ‘s/\r\n/\n/’ find . –name \*,v –exec perl –pi –e ‘s/\r\n/\n/’ “{}” \; d2u from Perforce Public Depot Solutions that don’t work! dos2unix …? Migrating: Example tony@barney:~/p4-migrate$ p4migrate -c cmap checkpoint.18

THIS SOFTWARE IS PROVIDED BY PERFORCE SOFTWARE, INC. 'AS IS' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL PERFORCE SOFTWARE, INC. BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Do you accept these terms and conditions? : y Opening input and output files... Starting scan of checkpoint file... Parsed 10k records (5000.0 records/sec) Writing conflict map... p4migrate warning:

Your checkpoint still contains inconsistencies. Review the conflict map, make any necessary adjustments and then retry. You should NOT use the edited checkpoint under any circumstances. Migrating: Example

#------# CONFLICT MAP GENERATED BY P4MIGRATE # The Rules: # 1. All mappings are 1:1 # 2. Later mappings override earlier mappings # 3. Left hand side represents existing (most common) case # 4. Change the right hand side to reflect the case you want # 5. Make case changes only # # Guidelines: # # 1. Put mappings for higher level directories above the mappings for their # subdirectories # 2. Add new mappings for every non-leaf directory whose case you change # 3. Be consistent! #------//depot/... //depot/... //depot/mac-dev/... //depot/mac-dev/... //depot/main/... //depot/main/... //depot/main/jam/... //depot/main/jam/... Migrating: Example tony@barney:~/p4-migrate$ p4migrate -t cmap checkpoint.18

Opening input and output files... Loading translation map from cmap Starting scan of checkpoint file... Parsed 10k records (5000.0 records/sec) p4migrate warning:

Your checkpoint is largely case-consistent, but still contains some minor inconsistencies. These will be resolved when you generate an edited checkpoint by re-running p4migrate with the -o flag. Migrating: Example tony@barney:~/p4-migrate$ p4migrate -t cmap -c cmap2 -o checkpoint.good checkpoint.18

Opening input and output files... Loading translation map from cmap Starting scan of checkpoint file... Parsed 10k records (10000.0 records/sec) Writing conflict map... p4migrate info:

Success: all case-inconsistencies have been resolved. Migrating: Example tony@barney $ p4d -r . -jr checkpoint.good Perforce db files in '.' will be created if missing... Recovering from checkpoint.good... Migrating: Example tony@barney:~/p4-migrate$ p4migrate -R -r . checkpoint.good

Opening input and output files... Starting scan of checkpoint file... Renaming depot/Main to depot/main Renaming depot/main/WWW to depot/main/www Renaming depot/main/www/Index.html,v to depot/main/www/index.html,v Renaming depot/main/JAM to depot/main/jam Renaming depot/main/P4-DOC to depot/main/p4-doc Renaming depot/main/Scripts to depot/main/scripts Parsed 10k records (10000.0 records/sec) p4migrate info:

Success: all case-inconsistencies have been resolved. Migrating: problems

Too many passes through checkpoint Slow for large sites Minimum two passes, often more Depot maps aren’t migrates in-flight c:\perforce\depot\… to /p4/depot/… Slows things down even more p4migrate – futures

Depot map translation Provide up-front mapping for your depot storage Archive migration With line-feed translation built-in No more need for tar + perl Will also work for ‘p4d –C1’ migrations Merging Perforce Databases Merging databases

A tour of the issues perfmerge2.pl perfmerge++ Merging: Issues

Colliding worlds Changelists and timestamps Files Users/Clients/Branches/Labels/Depots Typemap/Triggers/Groups/Protections Jobs & Fixes … It’s a jungle out there! Merging: Solutions Past perfmerge.pl Back in the dim and distant past Slow and very risky Merging: Solutions Past perfmerge2.pl Rewrite of perfmerge.pl Supported different merge types Slow and moderately risky Three types of merge: ‘merge’, ‘append’ and ‘offset’ Dictates whether and how changelists and jobs get renumbered Merge records optionally written to a ‘mapfile’ But often not… Merging: The merge types

‘merge’ Time-based Changelists are renumbered in time order ‘append’ Changes in second database ‘appended’ to the first. Breaks temporal nature of changelists ‘offset’ Second db appended to first, but numbers are ‘offset’ by a specified amount Merging: The present perfmerge++ Written entirely in C++ for speed Reads two sets of input db files Writes one set of output db files Supports all perfmerge2.pl merge types Version locked! Always use the right version! 2006.1 and 2006.2 builds available Merge record always saved to a file in the merged repository Merging: Example tony@barney $ perfmerge.062 -t merge ../left ../right .

Perfmerge++ is supplied by Perforce Software in the hope that it will be useful. It is a support utility, not a Perforce product. All use of this software is at the user's own risk and subject to the following terms and conditions.

Do you accept these terms and conditions? : y Running pre-merge sanity checks... perfmerge++ error: Domain conflict: both servers have a client/label/branchspec called 'bruno_ws'. Rename one of them and try again. Merging: Example tony@barney $ perfmerge.062 -t merge ../left ../right .

Running pre-merge sanity checks... perfmerge++ error: Jobspec conflict: both servers have jobs, but they do not share the same jobspec. Either resolve this situation, or delete all jobs from one of the servers before trying again. Merging: Example tony@barney $ perfmerge.062 -t merge ../left ../right .

Running pre-merge sanity checks... Pre-loading merge data... Loading counters... Loading and resequencing changelists... Loading and resequencing traits (attributes)... Loading and resequencing jobs... Merging... Saving merge details to changelist... Merge details saved to change 2000 Merging: Example tony@barney $ p4 describe -s 2000 Change 2000 by perfmerge++@perfmerge++ on 2007/03/12 17:02:50

The change below records the result of a merge of two Perforce databases performed using the utility perfmerge++. The file submitted contains a listing showing how the changelist and job numbers were renumbered as part of the merge.

This information may be useful in future.

Affected files ...

... //depot/perfmerge/map.txt#1 add Merging Example tony@barney $ p4 print //depot/perfmerge/map.txt //depot/perfmerge/map.txt#1 - add change 2000 (ctext)

Changes from ../left database Change 1 is now Change 1 Change 2 is now Change 3 Change 3 is now Change 5 … Changes from ../right database Change 1 is now Change 2 Change 2 is now Change 4 Change 3 is now Change 6 … Jobs from ../left database Job job000001 is now job000001 Job job000002 is now job000003 Job job000003 is now job000005 … Jobs from ../right database Job job000001 is now job000002 Job job000002 is now job000004 Job job000003 is now job000006 … Summary

Migrating Same platform is easy Windows to Unix with ‘p4d –C1’ is easy Windows to Unix with p4migrate is relatively easy Unix to Windows is extremely hard Merging perfmerge++ makes it look easy, but it’s not perfect Any Questions?