[Tes3cmd] Using Tes3cmd to clean plugins the easy way.

[Spirithawke]

You can setup a drag and drop batch file to use with Tes3cmd to easily clean any plugin (esp or esm) file.

Note: I am currently using Standalone tes3cmd.exe 0.37v (for Windows), tes3cmd-0.37v-2013.10.06.7z from MLOX Download page

This is some information culled from a TesPCD thread over on the official forums:  Q: Cleaning Mods with TESPCD
Original post by Pinkertonius

Pinkertonius, on 30 Apr 2011 - 4:50 PM, said:
I highly recommend using Tes3cmd, it will do a really through job of cleaning away the duplicate entries (and some other stuff).

Here's a little how-to threw together quickly a while back.

Show Spoiler

TES3CMD by John Moonsugar

This program is great at cleaning, fixit, fiddling with a mod. It's very thorough and catches stuff other cleaner do while not breaking the mod (Yes, I'm looking at you TESTOOL). It can update the headers to match the ESMS, recover bad mods, dump data, and so much more. It even has a "fixit" command that will actually fix it (unlike ...ahem cough..TESTOOL...cough).

The main drawback is no GUI. That does limit audience, because to be honest while versatile, the command line just isn't that much fun. Heck in windows, its a pain to even get it the directory you want. But I have a way:

Manual Method:
Find your Morrowind/DataFiles directory
Right click it while holding shift
Select "open cmd window here" and a Command box will appear with the directory all ready set to Data Files.
As long as TES3CMD is in that directory,
Type "TES3CMD clean <modname.esp>" or "Tes3cmd fixit" (to fix everything)

I also have made a bat file that will allow you to drag and drop an esp right on top of it and it will clean it then put the results and log file right back where the ESP came from.
I use this for my "Morrowind Archive" directory. I like to clean the mods before I put them in my Data Files directory. Here's how (sampled for TESELINT)

More "I hate command lines" friendly method:
Open Notepad. Cut and paste below. Save as .bat file where ever you installed TES3CMD. Then find the file, right click and select Send->Send to desktop as shortcut. Now you can drag an esp from any folder onto your desktop shortcut and it will clean it and put the clean one right back where it came from.

Batch file text (cut and paste the below)

Code: Select all

@echo off
rem drag and dropable bat file
rem put shortcut on desktop and drag esp onto it
rem "clean" file and log will be added to same folder the esp came from

tes3cmd clean %1 > %1-cleaned.log
type %1-cleaned.log
pause


In my experience 95% of my mods were somekind of dirty. And this fixes that.

TESPCD is good as a diagnotic tool but I don't directly clean with it . Also, I don't believe it updates the record count in mods that it edits. And I advise against using it to delete any dialogue. It doesn't "see" the PNAM and NNAM entries the surround the info entries.

Notes about this:

First, make sure the mod master dependencies are already located in the "\Data Files" folder, or THE MOD WILL NOT BE CLEANED !!

TO RUN...
Drag ESP/M files onto the shortcut or .bat file to clean mods.

CLEANING ORDER of mods...
1.  Mod masters (e.g. GIANTS.esm, etc.)
2.  Mod plugins that are a Master to another Plugin
3.  Mod plugins relying on Official Masters, Mod Masters, or Plugin Masters
4.  All other plugins without master dependencies

DO NOT CLEAN MOD "CONTROL FILES" ...
Files such as "GIANTS Ultimate Control File.esp" intentionally contain records duplicated from their mod master, and should not be cleaned.

tes3cmd.exe AND the batch file (whatever you name it, I named mine tes3cmd clean.bat) MUST be in the Data Files directory. However, the esp itself do not need to be in the Data Files directory. I have successfully cleaned esps in folders on my desktop (as long as all Master Files ARE in Data Files so Tes3cmd can find them).
The above batch file example leaves the old espname.esp and a new clean_espname.esp in the data files directory (or whatever directory you started from) along with the log file.  For myself I prefer to have it go ahead and move the old esp to the Data Files\tes3cmd\backups directory.  So my batch file looks like this:

Code: Select all

@echo off

rem drag and dropable bat file
rem put shortcut on desktop and drag esp onto it
rem the cleaned file with original name will be in Data Files with log file
rem (assuming that is where you started from).
rem the original will be in Data Files\tes3cmd\backups directory

tes3cmd clean --replace --hide-backups %1 > %1-cleaned.log 2>&1

rem Output results of tes3cmd cleaning to CMD window ...
rem Outputting results to file via "> %1-cleaned.log 2>&1" as above will not print to screen
rem without explicitly using 'type' command below ...

type %1-cleaned.log
pause

This version of the batch file does not synchronize the mod header to the esm files the mod depends on, so it may be yellow in Wrye Mash and give an in-game error until you fix it in Wrye Mash OR run Tes3cmd like this:
tes3cmd header --synchronize "my plugin.esp"

If you don't want it to pause in the dos window when it finishes, delete the pause
If you don't want it to write a log delete "> %1-cleaned.log 2>&1"
It can take quite a while for Tes3cmd to finish if the plugin is very large....BE PATIENT (one of my grass mods took 20 minutes and I have a fairly decent computer)
I will note that it is possible to make other types of drag and drop Tes3cmd batch files as well as this cleaning one by using different Tes3cmd switches from the Tes3cmd Wiki

IF you wish to clean Tribunal.esm and Bloodmoon.esm, DO NOT USE THIS BATCH FILE!!!  Use the fancy version down below.  It has special switch settings in it to leave the needed GMSTs alone.  Do not ever touch Morrowind.esm

I hope others find this information useful.

[Spirithawke]

A small update.  I finally worked my way through the entire reference thread, and I will post here the last working FANCY version of the bat file posted by RMWChaos: Post #97

[quote name='RMWChaos' timestamp='1306050073' post='17733492']

RMWChaos, on 22 May 2011 - 12:41 AM, said:
I revamped the code for the batch / cmd file, and tested it against some funky mod names, inlcuding this one:

Code: Select all

Test òñë (~`-!@#$%^&) 0987654321.esp


I removed much complexity, including the ability to make user-defined changes, which was just adding too many problems to the mix for what should be a very simple little batch file.

All seems to work appropriately; so give it a try and let me know what you think.

Code: Select all

@ECHO OFF

:START
CLS
REM *********************************************************************************
REM *****    "Drag-and-Droppable" TES3CMD Cleaning Batch / CMD File        *****
REM *****        by: RMWChaos    ver. 2011.05.22-02.33            *****
REM *****          original concept by: Pinkertonius            *****
REM *********************************************************************************
REM *
REM *    TO SETUP ...
REM *
REM *        1. Download TES3CMD v0.37o from:
REM *            http://mlox.googlecode.com/files/tes3cmd-0.37o-2010.08.20.7z
REM *
REM *        2. Unzip TES3CMD to your "\Morrowind\Data Files" directory
REM *
REM *        3. Create a new "T3Clean.txt" file in the same directory
REM *
REM *        4. Copy the entire contents of this code into the text file and save it
REM *
REM *        5. Rename the text file with a ".bat" or ".cmd" extension (either works)
REM *
REM *        6. *OPTIONAL* Create a SHORTCUT to the batch file on your desktop
REM *
REM *********************************************************************************
REM *
REM *    The cleaned plugin will take the place of the old 'dirty' plugin and the old
REM *        plugin will be backed up to "\Data Files\tes3cmd\backups" by default.
REM *
REM *    A log file called "<plugin.name>-cleaned.log" will also be created in the
REM *        same "\Data Files\tes3cmd\backups" folder by default.
REM *
REM *********************************************************************************
REM *
REM *    Tribunal.esm and Bloodmoon.esm will use a different cleaning method so as not
REM *        to remove required GMSTs, and tes3cmd may ignore (not clean) other mods
REM *         based on its own internal code.
REM *
REM *********************************************************************************

REM *********************************************************************************
REM *****            MOD CLEANING STEPS                *****
REM *********************************************************************************
REM *
REM *    First, make sure the mod master dependencies are already located in the
REM *        "\Data Files" folder, or THE MOD WILL NOT BE CLEANED !!
REM *
REM *    TO RUN ...
REM *        Drag ESP/M files onto the shortcut, .bat, or .cmd file to clean mods.
REM *
REM *    CLEANING ORDER ...
REM *         1. Official masters (i.e. Tribunal.esm and Bloodmoon.esm)
REM *         2. Mod masters (e.g. GIANTS.esm, etc.)
REM *         3. Mod plugins that are a Master to another Plugin
REM *         4. Mod plugins relying on Official Masters, Mod Masters, or Plugin Masters
REM *         5. All other plugins without master dependencies
REM *
REM *    DO NOT CLEAN MOD "CONTROL FILES" ...
REM *        Files such as "GIANTS Ultimate Control File.esp" intentionally contain
REM *        records duplicated from their mod master, and should not be cleaned.
REM *
REM *********************************************************************************


REM *********************************************************************************
REM *****    CHECK for OFFICIAL ESMs, INITIATE LOG, CLEAN, and SYNC        *****
REM *********************************************************************************

REM *    Set initial environment variables ...
SET MOD="%~nx1"
SET TMP=".%~nx1"
SET BAK="%cd%\tes3cmd\backups\.%~nx1"
SET DBL="%cd%\tes3cmd\backups\%~n1~*%~x1"
SET LOG="%cd%\tes3cmd\backups\%~nx1-cleaned.log"

REM *    Check for Morrowind.esm (cannot clean) ...
IF %MOD%=="Morrowind.esm" (
   ECHO You cannot clean %MOD%.
   PAUSE
   GOTO END
)

REM *    Initiate log file ...
ECHO Cleaning of %MOD% initiated on %DATE% at %TIME% > %LOG%
@ECHO: >> %LOG%

REM *   Append original mod header contents to log file ...
ECHO Pre-clean / pre-sync mod header contents . . . >> %LOG%
tes3cmd dump --type TES3 %MOD% >> %LOG%
@ECHO: >> %LOG%

REM *    Notify user that cleaning has begun ...
ECHO Cleaning of %MOD% has begun . . .
@ECHO:
ECHO You may not see activity for some
ECHO  time, especially on larger mods . . .
@ECHO:
@ECHO:

REM *    If file is NOT Tribunal or Bloodmoon ...
IF NOT %MOD%=="Tribunal.esm" (
   IF NOT %MOD%=="Bloodmoon.esm" (
       tes3cmd clean --hide-backups --replace %MOD% >> %LOG% 2>&1
       GOTO SYNC
   )
)

REM *    If mod IS Tribunal or Bloodmoon ...
ECHO "EVIL GMSTs" will not be cleaned . . . >> %LOG%
@ECHO: >> %LOG%

REM *    Temporarily rename mod to bypass tes3cmd no-clean behavior ...
REN %MOD% %TMP%
tes3cmd clean --hide-backups --replace --instances --cell-params --dups --junk-cells %TMP% >> %LOG% 2>&1

REM *    Rename mod and backup back to original name ...
REN %TMP% %MOD%
REN %BAK% %MOD%

:SYNC
@ECHO: >> %LOG%
@ECHO: >> %LOG%

REM *    Synchronize mod header ...
ECHO Syncrhonizing mod - update masters and header . . . >> %LOG%
@ECHO: >> %LOG%
tes3cmd header --hide-backups --synchronize %MOD% >> %LOG% 2>&1
@ECHO: >> %LOG%
ECHO Synchronizing complete. >> %LOG%
@ECHO: >> %LOG%

REM *    Append mod header description ...
ECHO Update header description . . . >> %LOG%
tes3cmd header --hide-backups --description "+ Cleaned by tes3cmd on %DATE% at %TIME%." %MOD% >> %LOG% 2>&1
@ECHO: >> %LOG%

REM *    Delete extra backups created during synchronization ...
IF EXIST %DBL% DEL %DBL%

REM *********************************************************************************
REM *****    TIMESTAMP LOG, OUTPUT LOG, and DISPLAY LOG LOCATION        *****
REM *********************************************************************************
:DISP

REM *    Append cleaned & updated mod header contents to end of log ...
ECHO Post-clean / post-sync mod header contents . . . >> %LOG%
tes3cmd dump --type TES3 %MOD% >> %LOG% 2>&1
@ECHO: >> %LOG%

REM *    Timestamp the log file with completeion date and time ...
ECHO Cleaning finished %MOD% on %DATE% at %TIME% >> %LOG%

REM *    Output log file to console ...
TYPE %LOG%
@ECHO:

REM *    Display location of log file and wait for user input to close CMD window ...
ECHO The log file for this cleaning process was stored here:
@ECHO:
ECHO %LOG%
PAUSE

REM *********************************************************************************
REM *****        End of batch commands, exit CMD window.            *****
REM *********************************************************************************
:END


Regards,

-RMWChaos

NOTE: The following things must be true for the above bat file to work:

Just to confirm, you should have the following in your /Morrowind/Data Files directory in order for this to work...

   tes3cmd.exe file
   /tes3cmd folder (this will be created the first time you run tes3cmd.exe or the batch/command file
   T3Clean.bat -OR- T3Clean.cmd with the batch code from this thread pasted into it
   The ESP you intend to clean along with any of its required Masters

-RMWChaos

Special Note: With Windows 7 the esp itself do not need to be in Data Files, it can be anywhere, only the required Master Files (esm files) MUST themselves be in Data Files so Tes3cmd can find them.

As long as the above conditions are met...you can use this as a drag and drop batch file by sending the shortcut to your desktop (as instructed in the first post) and then dragging the esp files from your data files folder and dropping them on the shortcut.  I am currently using a slightly modified version of this one for myself.  You can name the batch file whatever you want, such as "tes3cmd clean fancy2.bat" and it will work for you

Special Note:  The above batch file has code in it for cleaning Tribunal.esm and Bloodmoon.esm files.  IF you do this, you must re-synchronize ALL other mod files that depend on those 2 esm files.  Additionally I would not use this installation for releasing mods as the mods created using these modified esm files will before releasing need to be put into a setup where Blodmoon.esm and Tribunal.esm have NOT been cleaned and then re-synchronized, so that they show up correctly to the majority of users.  This would also apply to a mod that depends on some other cleaned esm.

[Spirithawke]

To clean all your mods at one time Xar recommends this:

Yeah, ok. It's easy, you throw the Tes3CMD into the data files folder, you right click on your Data Files folder while holding shift, open command window, and type "Tes3cmd fixit" (do NOT type the quotes in the cmd window) and all your mods will be cleaned.  IIRC, it worked for me.

This is what the Tes3cmd wiki says about the fixit option:

Show Spoiler

The fixit command does the following operations:
- Cleans all your plugins ("tes3cmd clean")
- Synchronizes plugin headers to your masters ("tes3cmd header --synchronize")
- Generates a patch for merged leveled lists and more ("tes3cmd multipatch")
- Resets Dates on Bethesda data files ("tes3cmd resetdates")

If you have a lot of plugins...be prepared for a very long wait, especially if some of them are very large...like say a grass plugin (which should NOT have been checked in the launcher to start with, and I have seen elsewhere recommended to NOT clean due to that long wait and the fact that you do not use them in game and MGE/MGEXE do not care about GMSTS and stuff when creating distant land).  NOTE: Doing it this way with no options specified fills your data files directory with esp files ending in ~1.esp and perhaps ~2.esp (if the esp needed to be synchonized).  These are the backup files.  This way also does not generate log files, so you have no history of what was changed in the esp files.  It also generates multipatch.esp which according to the Tes3cmd Wiki:

Show Spoiler

The multipatch produces a patch file based on your current load order to solve various problems. You should regenerate your multipatch whenever you change your load order. The goal of the "multipatch" command is that it should always be safe to use it with no options to get the default patching behavior (if you do find any problems, please report them and they will be fixed ASAP). When no options are specified, the following options are assumed:

 --cellnames --fogbug --merge-lists --summons-persist

The different patching operations are explained below:

Cell Name Patch (--cellnames)

 Creates a patch to ensure renamed cells are not accidentally reverted to   their original name.

 This solves the following plugin conflict that causes bugs:
 * Master A names external CELL (1, 1) as: "".
 * Plugin B renames CELL (1, 1) to: "My City".
 * Plugin C modifies CELL (1, 1), using the original name "", reverting renaming done by plugin B.
 * References in plugin B (such as in scripts) that refer to "My City" break.

 This option works by scanning your currently active plugin load order for cell name reversions like those in the above example, and ensures whenever possible that cell renaming is properly maintained.

Fog Bug Patch (--fogbug)

 Some video cards are affected by how Morrowind handles a fog density setting of zero in interior cells with the result that the interior is pitch black, except for some light sources, and no amount of light, night-eye, or gamma setting will make the interior visible. This is known as the "fog bug".

 This option creates a patch that fixes all fogbugged cells in your active plugins by setting the fog density of those cells to a non-zero value.

Summoned creatures persists (--summons-persist)

 There is a bug in Morrowind that can cause the game to crash if you leave a cell where an NPC has summoned a creature. The simple workaround is to flag summoned creatures as persistent. The Morrowind Patch Project implements this fix, however other mods coming later in the load order often revert it.
 This option to the multipatch ensures that known summoned creatures are flagged as persistent.

So you may as well keep and use the multipatch file.  Remember that if you change your load order, the multipatch file needs to be regenerated with tes3cmd multipatch from the cmd line.

EDIT: This is MY preferred way to use it for leveled list merging:  Simple bat file for multipatch that generates a log file so you can see what was done later (as always, just copy this to notepad and save it as an .bat file):

Code: Select all

tes3cmd multipatch > Tes3cmd_Multipatch.log
type Tes3cmd_Multipatch.log
pause

I saved it in Data Files as Tes3cmd Multipatch.bat so that it groups with Tes3cmd.exe.  Double click it any time you need to re-do to merged lists. Simple and easy.

[Spirithawke]

When I am just curious, I use this simple version to see if a mod is clean. It leaves all 3 files (cleaned esp, backup, and log file) in the Data Files directory (or whatever directory your esp started in):  tes3cmd clean mod checking.bat

Code: Select all

@echo off
tes3cmd clean --replace %1 > %1-cleaned.log 2>&1
type %1-cleaned.log
pause

I have went to using this version of the fancy bat file as I have rem'd out where it writes the cleaned date to the mod headers along with changing it to leave the log file in data files so I can easily look at it (I have N++ set to use for .log files in windows so is simple double click).  Once I am done with it I put it in a directory I made to save log files if it is going to be a mod in my load order.

Code: Select all

@ECHO OFF

:START
CLS
REM ***************************************************************************************
REM *****       "Drag-and-Droppable" TES3CMD Cleaning Batch / CMD File                *****
REM *****        by: RMWChaos                                                         *****
REM *****        original concept by: Pinkertonius                                    *****
REM *****        modified by Spirithawke ver. 2013.03.29                              *****
REM ***************************************************************************************
REM *
REM *   TO SETUP ...
REM *
REM *           1. Download TES3CMD v0.37u from:
REM *                   http://mlox.googlecode.com/files/tes3cmd-0.37u-2012.12.10.7z
REM *
REM *           2. Unzip TES3CMD to your "\Morrowind\Data Files" directory
REM *
REM *           3. Create a new "Tes3cmd clean fancy2.txt" file in the same directory
REM *
REM *           4. Copy the entire contents of this code into the text file and save it
REM *
REM *           5. Rename the text file with a ".bat" or ".cmd" extension (either works)
REM *
REM *           6. *OPTIONAL* Create a SHORTCUT to the batch file on your desktop
REM *
REM *********************************************************************************
REM *
REM *   The cleaned plugin will take the place of the old 'dirty' plugin and the old
REM *           plugin will be backed up to "\Data Files\tes3cmd\backups" by default.
REM *
REM *   A log file called "<plugin.name>-cleaned.log" will also be created in Data Files
REM *
REM *********************************************************************************
REM *
REM *   Tribunal.esm and Bloodmoon.esm will use a different cleaning method so as not
REM *           to remove required GMSTs, and tes3cmd may ignore (not clean) other mods
REM *            based on its own internal code.
REM *
REM *********************************************************************************
REM *
REM *********************************************************************************
REM *****                       MOD CLEANING STEPS                              *****
REM *********************************************************************************
REM *
REM *   First, make sure the mod master dependencies are already located in the
REM *           "\Data Files" folder, or THE MOD WILL NOT BE CLEANED !!
REM *
REM *   TO RUN ...
REM *           Drag ESP/M files onto the shortcut, .bat, or .cmd file to clean mods.
REM *
REM *   CLEANING ORDER ...
REM *           1. Official masters (i.e. Tribunal.esm and Bloodmoon.esm)
REM *           2. Mod masters (e.g. GIANTS.esm, etc.)
REM *           3. Mod plugins that are a Master to another Plugin
REM *           4. Mod plugins relying on Official Masters, Mod Masters, or Plugin Masters
REM *           5. All other plugins without master dependencies
REM *
REM *   DO NOT CLEAN MOD "CONTROL FILES" ...
REM *           Files such as "GIANTS Ultimate Control File.esp" intentionally contain
REM *           records duplicated from their mod master, and should not be cleaned.
REM *
REM *********************************************************************************
REM *
REM *********************************************************************************
REM *****       CHECK for OFFICIAL ESMs, INITIATE LOG, CLEAN, and SYNC          *****
REM *********************************************************************************
REM *
REM *   Set initial environment variables ...
SET MOD="%~nx1"
SET TMP=".%~nx1"
SET BAK="%cd%\tes3cmd\backups\.%~nx1"
SET DBL="%cd%\tes3cmd\backups\%~n1~*%~x1"
SET LOG="%cd%\%~nx1-cleaned.log"

REM *   Check for Morrowind.esm (cannot clean) ...
IF %MOD%=="Morrowind.esm" (
       ECHO You cannot clean %MOD%.
       PAUSE
       GOTO END
)

REM *   Initiate log file ...
ECHO Cleaning of %MOD% initiated on %DATE% at %TIME% > %LOG%
@ECHO: >> %LOG%

REM *   Append original mod header contents to log file ...
ECHO Pre-clean / pre-sync mod header contents . . . >> %LOG%
tes3cmd dump --type TES3 %MOD% >> %LOG%
@ECHO: >> %LOG%

REM *   Notify user that cleaning has begun ...
ECHO Cleaning of %MOD% has begun . . .
@ECHO:
ECHO You may not see activity for some
ECHO  time, especially on larger mods . . .
@ECHO:
@ECHO:

REM *   If file is NOT Tribunal or Bloodmoon ...
IF NOT %MOD%=="Tribunal.esm" (
       IF NOT %MOD%=="Bloodmoon.esm" (
               tes3cmd clean --hide-backups --replace %MOD% >> %LOG% 2>&1
               GOTO SYNC
       )
)

REM *   If mod IS Tribunal or Bloodmoon ...
ECHO "EVIL GMSTs" will not be cleaned . . . >> %LOG%
@ECHO: >> %LOG%

REM *   Temporarily rename mod to bypass tes3cmd no-clean behavior ...
REN %MOD% %TMP%
tes3cmd clean --hide-backups --replace --instances --cell-params --dups --junk-cells %TMP% >> %LOG% 2>&1

REM *   Rename mod and backup back to original name ...
REN %TMP% %MOD%
REN %BAK% %MOD%

:SYNC
@ECHO: >> %LOG%
@ECHO: >> %LOG%

REM *   Synchronize mod header ...
ECHO Syncrhonizing mod - update masters and header . . . >> %LOG%
@ECHO: >> %LOG%
tes3cmd header --hide-backups --synchronize %MOD% >> %LOG% 2>&1
@ECHO: >> %LOG%
ECHO Synchronizing complete. >> %LOG%
@ECHO: >> %LOG%

REM *   Append mod header description ...
rem ECHO Update header description . . . >> %LOG%
rem tes3cmd header --hide-backups --description "+ Cleaned by tes3cmd on %DATE% at %TIME%." %MOD% >> %LOG% 2>&1
rem @ECHO: >> %LOG%

REM *   Delete extra backups created during synchronization ...
IF EXIST %DBL% DEL %DBL%

REM *********************************************************************************
REM *****       TIMESTAMP LOG, OUTPUT LOG, and DISPLAY LOG LOCATION             *****
REM *********************************************************************************
:DISP

REM *   Append cleaned & updated mod header contents to end of log ...
rem ECHO Post-clean / post-sync mod header contents . . . >> %LOG%
rem tes3cmd dump --type TES3 %MOD% >> %LOG% 2>&1
rem @ECHO: >> %LOG%

REM *   Timestamp the log file with completeion date and time ...
ECHO Cleaning finished %MOD% on %DATE% at %TIME% >> %LOG%

REM *   Output log file to console ...
TYPE %LOG%
@ECHO:

REM *   Display location of log file and wait for user input to close CMD window ...
ECHO The log file for this cleaning process was stored here:
@ECHO:
ECHO %LOG%
PAUSE

REM *********************************************************************************
REM *****               End of batch commands, exit CMD window.                 *****
REM *********************************************************************************
:END