Terry's TMG Tips

Splitting Lumped Sources

This page created 24 Apr 2011

Version note: Applies to TMG 8 & 9

This article describes how to convert from so-called "lumped" Sources to the "split" style. For example, one might have created a single Source in TMG for all census records in a given county and now wants to convert to a system with one Source for each household. Or one has created a single Source for all e-mails from a correspondent, and now wants to have one Source for each e-mail. My article on Using the Split CD to “Lump” Sources describes one way of creating such "lumped" Sources. This article can help if one has used a "lumped" method and has now decided, as I have, that converting to more discrete Sources is preferable.

There are various methods for making such a conversion. The method I describe here uses John Cardinal's TMG Utility to significantly reduce the effort required. John's program is designed to assist in many tasks in managing your TMG Project, including the one described here. The program is "donorware," meaning it is free. But if you find it useful, John asks that you make a donation in his name to his favorite charity. I would suggest that if you have made a donation in the past and still find TMG Utility is helping you in the use of TMG, especially if you are making use of features such as the one described here which have been added by recent updates, that you consider making a new donation.

The Conversion Process

The basic process in converting a group of Source from "lumped" style to "split" style is straightforward:

1. Identify the Source to be split.
2. Create new Sources for each of the portions of the existing Source you want to now stand alone. For example, if you have one Source for the census in each county, you might what to create a new source for each household you cited. Typically the details needed to create the new Sources will have been recorded in the Citation Detail (CD) of the various Citations to the old Source.
3. Find each citation that is to be moved to the new Source, change it's Source Number, and remove the information in CD that has been transferred to the new "split" Source, while retaining any notes about that source which are specific to a given Tag.

This last step can be daunting. If there are only one or two citations to be transferred to the new Source, doing this manually in TMG is probably the easiest method. But if there are many, as is often the case with census Sources, for example, doing this manually will be a massive task. The method described here substantially reduces that effort. Nevertheless, converting a substantial number of Sources will take some time, and requires concentration and focus to avoid introducing errors into your database.

The process is described below in more detail.

Creating a Checklist

I find it essential to create a checklist of the Sources and Citations to be converted. The best way I've found to do that is to use the List of Citations report, sent to a text file, to use as a working checklist. Having the list in a text file enables you to copy text strings and paste them into fields in TMG Utility during the process. I create my check list with a List of Citations report using this filter:

The term "[?]" for the value, rather than entering the Source Number directly, causes TMG to prompt you for the Source Number each time the report is run, thus avoiding the need to change the Filter each time the report is run for a different Source. I've found the Indented List Type (under Options) to work best, as it sorts the list by the contents of the CD field. If your CD entries contain the principal information (such as name of head of household in a census source) first, the groups the citations as they will be used for the new "split" sources. Here is an example:

For use as a checklist, note that I made several edits to the report after it was created by TMG, as marked by the red arrows in the screenshot above:

1. I set off the groups of citations to be converted to each new Source by adding blank lines and strings of equal signs. I identified the groups by examining the name of the subject in the CD text, in this case the subject of a draft card.
2. As I added the new Sources in TMG, I recorded the new Source Numbers in the list for use during the conversion process.
3. Some of my CDs contained strings long enough to wrap to a second line in the report. Since I would be copying a portion of the CD which is to be removed into TMG Utility, I removed the return and added back the space that had been removed by the report to the first Citation in each group.

After testing one or two conversions to be sure I understood the process, I found it helpful to work the process in batches. I created a number of new Sources as illustrated in the screenshot above, then converted that group before making more new Sources.

Use TMG Utility to Transfer Citations to their New Sources

The actual transfer of Citations to new Sources is in theory simple: find the Tag with the Citation, open the Citation, change the Source number, and delete any text in the CD that is no longer needed. The problem is that with potentially hundreds of Citations to modify, this would be a very time-consuming and error-prone process if done manually. The use TMG Utility makes the process considerably easier and more dependable.

If you have not done so previously, first download and install the TMG Utility.

Important: The changes described here are "major surgery" to your Project. Errors are possible. Make a backup of your Project before you begin.

Your Project cannot be open in both TMG and the TMG Utility. Before starting the Utility, close your Project in TMG, using the File > Close Project menu command.

Setting Up the Change Citation Parts “Rules”

Open TMG Utility, and use the File > Open menu command to open your Project in the Utility. Then select Citations > Change Citation Parts in the left panel (version 7.1 or later) to see this:

This function of the Utility requires the user to define "Rules" which tell program which Citations to change, and then what changes to make. As we will see below, defining these Rules is not difficult. Click the Set Rules... button to open the Rules Editor:

Note that there are two parts to the Rules: Filters, which tell the Utility which Citation to change; and Actions, which tell it which changes to make. You create Rules by defining them with the fields below each of the two sections. Select a data element from the first drop-down list, then select an operator from the second drop-down. Finally, enter a value in the third field. In the screenshot above, I have defined the "old" Source Number whose Citations are to be changed. Once the term had been defined, enter it in the list by clicking the Add Filter or Add Action button.

If you need to edit an existing Rule term, select it and make the necessary changes. Then click the Update Filter or Update Action button to modify the term.

I found the following Rules, illustrated in the screenshot above, worked well for converting my lumped Sources:

• Filter for Source Number: If Source Number   equal to   786
  Number 786 is the Source I am trying to split. Since I would be splitting a number of Sources off this one old Source, I found it useful to enter the Source Number directly in the Filter.
• Filter for CD text: If Detail   contains   [V1:CD text string]
  This term finds a specific group of Citations that are to be converted to a given new Source by examining the details that were entered in the CD of those Citations. I could have entered the actual text string in the Filter, but it changes for each new Source. So I used a variable instead, which causes the program to prompt me each time the program is run, making it easy to move to a new group. The variable is entered by typing the "["character, the letter "V," and a number between 1 and 9, followed by the "]" character. I added the ":CD text string" element so the program will remind me what data I should enter when prompted.
• Action to change Source Number: Change Source Number   set to   [V2:New Source No]
  This Action tells the program what new Source Number to assign to this group of Citations. Since each group will use a different new Source, I again used a variable, this time with the label New Source No.
• Action to remove CD text: Change Detail  replace   [V1]
  Since the details contained in the CD are now contained in the Source Definition, I wanted to remove them from the CD. I used the "replace" operator, and since the entire string of text to be removed was already entered in the Filter element above, I used the same variable, [V1]. Since the prompt text for this filter was defined the first time it appears, there is no need to repeat it. The "replace" operator also provides a "with" field for the text that is to replace the specified text. Since I was just removing text, not adding anything in it's place, I left that field blank. Note that if the CD contains other text, the operation will leave that text unchanged.
• Action to remove final ||: Change Detail   replace   ||
  In the Split CD construct I used, any note about the specific Citation appears in the last CD segment. If there there is an entry in that segment, it is separated from the prior text with a double bar code. But if that segment is left empty, this final double bar code will not exist in that Citation. So I omit that final double bar in my search string, and remove it from the CD if present with this Action term. If it doesn't exist in a given Citation, this term is simply ignored.

Close the Rule Editor when you have finished with the creation of Rules.

Applying the Changes

Once the Rules are defined we are ready to proceed with modifying the groups of citations. We now can see the Rules we created in the Rules box on the main screen of the Utility:

To proceed with the first set of changes, make sure the "Log Only" box is checked. This allows you to see a log of what changes will be made so you can verify that they are correct before any actual changes are made. Then click the "Change Citation Parts" button to start the process. Since I used variables in the Rules, I was then prompted to enter text for those terms:

The first prompt is for the CD text to be used for the search Filter and also for the replace Action. I copied that from the checklist, as shown below.

Note that this entry does not include the final double bars and additional note, unlike the following one, which continues "...||shows date, age 27..." Since I remove that final double bars in a separate Action, I am careful to not include it in this text string.

Click the OK button on the Set User Variable 1 dialog box, and a second prompt will appear for the new Source Number. Enter that, and the Log will appear, with the notation "No changes were made because the Log Only option is selected" at the bottom.

Check the log carefully to insure that the desired changes are being made to the CD text, and that the correct new Source is specified. Compare the number of Citations to be changed with the number shown in the checklist. If there are fewer on the log than on the checklist, that generally means that some of the citations have a subtle difference in the CD text. Find them and mark them so they can be addressed in a separate run with a modified search string.

Once you are satisfied that all is well, uncheck the "Log Only" box, and click the "Change Citation Parts" button again. The same prompts appear, populated with your previous entries. Click the OK button on each prompt to accept those same entries. The changes will be made and log will be updated (removing the notation that no changes were made).

Processing Additional Groups of Citations

Initially it is best to open the Project in TMG and verify that the changes were made as expected (no need to close the Utility; just remember to close the project in TMG before making another run in the Utility). Once you have verified that the process is working as expected, you can make a group of changes without checking each one in TMG.

To change additional groups of Citations use these steps:

• Verify that the Project is still closed in TMG if you have opened it to check results or create new Sources.
• Make sure the "Log only" box is checked.
• Click the "Change Citation Parts" button.
• Copy the new search text from the checklist to the prompt, and click OK.
• Update the new Source number for the new group, and click OK.
• Review the log to verify the changes to be made are correct.
• If not, repeat the steps above, making necessary corrections.
• Uncheck the "Log only" box.
• Click the "Change Citation Parts" button.
• Click OK on the dialogs for both prompts.

These steps quickly become rote, enabling a large number of changes to be processed in a reasonable amount of time. It is essential to take care to do each one correctly to avoid errors, as it is easy to miss one and potentially create a substantial number of erroneous entries.

Copyright 2000- by Terry Reigel