.LOG - notes on webwork [sheba] - working on new bonap file

8:44 AM 5/13/96-looks like good match BUT both combo fields omitted in the new file? - so, will delete current bonap96a, impress again and sent query to amy wrt lost fields

9:34 AM 5/13/96-tracyin3.ipf and txcenzi3/txcenpl.ind taken over to the tracy and tested a bit - changed row#=1 for county listing in that 1st of series was not on the selection menu and got no errors - copied that version back here - WRT above, Amy says both combo fields no longer maintained by them - so will have to revise all output programs that used these by constructing equivalent string from base data - BUT prior attach problems are probably due to a non-conforming kman base table and will start over again - with dbf - to check on this

9:44 AM 5/13/96-impressed bonap96a.itb from kartesz3, deleted both combo fields and added parents str 200 and qfield str 3 - see how this works

9:46 AM 5/13/96-worked fine using pdox text export with all fields delimited - even took rankcode as a numerical field - so, now ready to do quite a few things, 1st item is to extract a new karpoa1

10:04 AM 5/14/96-diverted to setting up final 'base' verison of the endemics and looking at how the 'dynamic' version might be formed - also - on p100 - working up a communications page for the FTC - now going over to the tracy to see if trixie can be linked to the bio network

2:48 PM 5/14/96-got things fixed at the tracy - still hacking at the ftc page

4:24 PM 5/14/96-sheba screen has gone south - something serious i think - might have to take it in

9:35 AM 5/15/96-took sheba to computerland and problem - weird lines images and lightening of text - did not happen and all is ok now - might be a heating probem - will see if it recurrs - have final edits on endemics tpwwwoa.ipf output done and copied 'background' files here [sheba] from server for some touchup - will put all in ftc dir and announce to ftc soon. - editing these in notebook

10:16 AM 5/15/96-had to take tp1del.htm to word to deal with italics - sent to ann to finish-sheba screen has faded again - will have to get fixed.

10:30 AM 5/15/96-also, not enough mem to do tp1lit.htm in notebook - will do in word

10:01 AM 5/16/96-new aspt pages sent to server with no comment yet - also set new endemics pages in ftc dir with pointer on the mailserver - now going to clean up the office

12:27 PM 5/16/96-visit to tracy - someone actually inputting stuff - to avoid cleaning office - will take a look at the fws endangered table/listings to update to the feb 96 file - have this here, but all other stuff on the p100 - have 2.5 mg of "fw*.*" in kman\herb - looking at what i have down

2:45 PM 5/16/96-visited with erich and discussed dynamics of mg - know a bit more about the process and will interact to generate the aspt search thing

3:18 PM 5/16/96-fwswwwo.ipf uses cronq3 and fwslist1/fws1date to produce the single file

3:23 PM 5/16/96-have candidates and those dropped but no endangered - will go back and get: - they now have a listing - current as of apr 96 - BUT looks like the same data file [oct 95] is available - might want to dump the apr 96 listing and match against current [oct 95] data, BUT looks like i should focus on the candidate stuff for now - thus, need to look at how this was done with past data - single table - and modigy to deal with two tables - listed candidates and those rejected

4:25 PM 5/16/96-so, downloaded the endangered list - fwsend96.txt to my docs/fws96 and will now transfer all relevant fws files from the p100 to either my docs [none kman] or kman/fws - candidate are dealt with by fwswwwo2 and fwslist2.itb - all need some work and can take a look at bouncing these names, candidates and listed, against bonap96a - copied all

7:35 PM 5/16/96-looks like both listings - 96cnort1 and 2 are same structure but the latter has an extra 'explanation' field -  and the former similar to fwslist2.  So, will set both up as different tables - 96cnort1.doc [candidates] will edit to fwsca1a.doc/txt and 96nort2.doc to fwsca2a.doc/txt with appropriate tables - will trim italics codes from both to allow bonap bounce - codes seem to be defined in 96cnortx.asc - so - fwsca1a = table 1, adn fwsca2a.doc = table 2 - full titles are in the original files - have all vascular plants - have 297 in table 2 [replacing # and ^]

9:03 PM 5/16/96-progressed to end on both and discovered that could not use 'find' to find a paragraph [^p] BUT works ok with orig ascii files - has something to do with 'default paragraph' in template i think - anyway, will have to start again - problem occurs when file is saved as a word doc - am SURE it has something to do with 'template' SO will keep files as text - NOW when i try the replace, it adds another ^p - will have to check notes from the p100 to see what i did with this

7:53 AM 5/17/96-opened the fws asc files in p100 word and section breaks evident - saving to doc file does NOT insert invisible ^p on p100 - surely something to do with the 'normal' template on the p100 version of word - for now will do edits and attaches on the p100 - took all files fws*.* in kman\herb on the p100 to kman\fws - will keep new names and indicated above - impressing fwslist2 to fwsca1a.itb - changing:

lead_reg from str 1 to str3 and deleting catecode [last field]
trendcode to priority str 3 [from str 1] - a few problems attaching - one of the replaces took a ^p from some records and these are marked by a " in the category field - will fix - did this with a replace and this left hrange field without a closing " - produced error messages on attach but table looks ok - still have some periods at the end of hrange 
10:42 AM 5/17/96-more tinkering - still get 12 'expected close quote' message on attach, but this might be empty lines at the end that i can't delete - so will go with this table - fwsca1a.itb - 348 records - now for the 2nd fws table - looks like replacing the section breaks with nothing is a problem - will replace with a ^p - NO this is not the problem - problem IS that many lines in file NOT separated by a ^p - got a lot of them with global replaces, but a lot remain - will continue when back from dallas

7:47 AM 5/20/96-hacked at getting fwsca2a.doc in shape and ready to start the attaching process - this one also has 7 chars, so will impress fwslist2 to fwsca2a.itb, delete last field 'catecode', and rename/resize fields - actually, will impress from fwscaa.itb which has most of this done - attach worked on 1st try - 281 taxa, including syns - have, in terms of status:

e - 108 - taxa listed as endangered
r - 104 - no support for listing
t - 11 - listed as threatened
y - 50 - synonyms

wrt 'explaination;

blank - 58
A - 65
I - 25
L - 119
N - 11
X - 3

9:07 AM 5/20/96-checked last endangered listed downloaded from fws and it has a later mod date - 12/2/96 - than prior - will bring this file into word on the p100 and save as fwsen1a.doc, impress old table - fwslist1 to fwsen1a.itb and, once attached, check to two listings - once this done - will see about comparing all 3 tables to the new bonap listing - fwslist1.itb has html italics codes in sciname field - will delete these so comparison with bonap can be done, but will have to edit output files to allow insertion during output

9:35 AM 5/20/96-quite a few with no paragraph breaks - doing global insertion of ^p based on "na","na" " - but this will probably not work for all - actually did replace for " "# with " ^p"# and got over 260 - parenthetical statements within the sciname field are a problem, esp. WRT the bonap check - think will try to fix this AFTER attachment, maybe by deriving two fields from sciname, comments and sciname1 

8:30 PM 5/20/96-hand-edited parenthetical comments from all new fws tables-looking at new mexico files for match - structure difference - converted bonap96a to b - will take out family and genus records = still 65k records - indexed bongen1b.ind by genus

9:14 AM 5/21/96-trimmed all remaining parenthetical comments from sciname in all fws tables - now looking at converting sciname string to something that can be used to search the new bonap table - looks like all variants in fws tables have either var. or ssp. and all with only 1 infra specific rank - so, can match(sciname," ") to get genus name and use match var./ssp. along with len(sciname) to break out other names - will move fwsen1a over the bonap96 to use bonap96b and base things on the new mexico checker

2:54 PM 5/21/96-have sciname broken down to search strings - will now convert fwsen1a to fwsen1b.itb and add fields to be extracted from bonap96b.itb - both id codes, author

3:02 PM 5/21/96-checked author in bonap96b - have 3 longer than 60 chars and none longer than 70 - these 3 look truncated - check the orig when possible - 'C. Christens.' and 'non-leymus'

9:03 PM 5/21/96-ran fwskar1.ipf - have 20 in fwsen1b.itb to fix - mostly species names

9:17 AM 5/22/96-going to add bonap family to check for same family placement BUT re-run takes too long due to full table check for not found - thinking of changing 'obtain next' with while loops that check each record of a given genus name for species/infra and stop when next genus is encountered - this might be possible in that bonap96b.itb is sorted by family,genus,species, etc.

9:41 AM 5/22/96-diverting from this to set up a routine that replicates karpoa1 from bonap96a - will need to create a combo2 string

1:01 PM 5/22/96-have 5204 records in bonap96a.itb for grasses - this includes genus name records - 4681 from bonap96b - 4813 in karpoa, 4565 iwth rankcode gt 1 and this is the record number for karpoa1.itb, the table now used by tracyin3.ipf - copied addstuff.ipf from tracy to bonap96 - will edit to addstring.ipf and set up combo2 and pluckstr for grasses from bonap96b, which will be called karpoa2.itb [will change table names in tracyin3.ipf].

7:41 PM 5/22/96-stringad.ipf works fine-now have fresh table for the tracy - will ship over tomorrow

10:26 AM 5/23/96-had to ding with tracyin3 here - was getting array errors, then over there - was not listing 1st county from county on label - changed single row#= (see !!!check in file) and seems to work over there now - saved functional version from over there as tracyin3.arc on trixie - they have put in 178 records - i marked test records with 'test' in the accession # field - thinking now that we need to use this to build a collector name lookup and add this, plus collector# and date to current sweep - also downloaded the texas.gz file from usgs and used winzip [installed yesterday] to extract it - no problem - but no delimiters - will see if pdox will bring it in as columned

10:49 AM 5/23/96-tried pdox and klutzy, then tried excel - import nice by record# limited to 16384 - will try pdox again - move base file from c:\stuff to my docs\usgs and will check their pages for info on the file - their readme file says:

The format of State and Territory files is:
 
     Columns               Contents
     1-100               Feature Name
     101-109             Feature Type
     110-144             County
     145-155             State/County FIPS Code
     156-170             Geographic Coordinates
     171-185             Source Coordinates of Linear Features
     186-190             Feature Elevation
     191-240             Topographic Map

the usgs file extracted is dated 12/8/95 and called texaftp - brought this into word and saved from word as text file to usgstxa.txt - this after a pdox import - using the wizard - produce a corrupt file after the heading error - will try again

12:44 PM 5/23/96-worked and maybe required a word text file - BUT allowed a break of the 1st field suggested by pdox this is not real - so will try one more time to get a good table

1:10 PM 5/23/96-this worked - 59217 records - now to configure as a lookup to replace the census table - BUT - also need to work on revision of texas grasses/bonap grasses - as well as fws stuff - will finish up the fws stuff 1st and try to set up fwskar1 to work better on the taesall4 problem

6:52 AM 5/24/96-sheba screen was unworkable last night - opened fuzzy and got worse with two tries - BUT is working fine today - backed up both new bonap tables to FLORA/kartesz - should have the original db file over there somewhere - using bio net to ship working files to appropiate places on the p100 - set up a base 'sheba' dir on the p100 and copied files from aspt/nm/fws into the base of this but kept contents of other kman folders intact to this - doing the same for folders in sheba my documents - going to p100 base folder shebamd - choked on shipment of usdatexa - will check - copied only most recent file

10:25 AM 5/25/96-sheba screen went to hell later in the am yesterday - took to computerland, but failed to go bad there - so hanging on till it gets worse - have a backup on of this file on bio/common/flora and other backups on sheba dirs on the p100 - now working on fwskar1a using elements of nmkar1 and 2 - trying to make a FIRM program for BONAP names checking that can be used for other listings - version 'a' uses obtain while loops to avoid full table checks from 'obtain next' statements - am now adding a 'namefix' field to fwsen1b that will carry the name as it occurs in the bonap data - will be different from fws sciname - NOW WORKING ON FWSFORM2.IPF

10:21 AM 5/28/96-fwskar1a.ipf works pretty good and have all that are covered by BONAP - eddy now on board and must divert to questions of how to deal with images that he will be producing - thinking that we need an updated [bonap96] listing of taesall4/bonap to include bonap id codes for both www and hardcopy - will see what can be done - thinking that can use bonap 7 char string to id taxa and one other to denote type of image

7:46 AM 5/30/96-diversion to endemics - decided to retract offer to FTC - lack of response - and carry it here - will modify tpwwwoa to tpwwwob.ipf to generate non-FTC pages that assume a functional annotation system [erich now working on it] - eddy now at tracy, calvin getting op sys switch from win3.1 to winNT

8:34 AM 5/30/96- took all gif/jpg from root/webwork and same + psd from photoshp/tfpimage

8:12 PM 5/30/96-set up tpwwwob.ipf as functional and generating new html files [as opposed to prior version which will retain for FTC] - now need to set up a page that deals with annotations - endemann.htm

also downloaded collector.zip from the harvard ftp, took to harvcol1.doc as a word file and output from word as a text file - 10 fields delimited by $ - now see ing pdox will import it - wizard choked again, but straight import no problem BUT have only 5 fields, last record = 47916 - fields

ref# 5
Name 144
date 43
labname 133
source 154 - looks like the import took only the 1st 5 fields

1.  Name         :  the name of the collector(s) in the form last
                  (family) name, first name, middle name, written as
                  fully as known.
2.  Dates        :  the birth and death dates, collection dates (col.),
                  years when the person was most active (fl.), or year
                  when the specimens were received by the herbarium
                  (r.).  x = year unknown.
3.  Abbrev.      :  the abbreviated form of the collector(s)'s name,
                  usually as it appears on specimen labels.  The family
                  name (last name) is always written in full, e.g. S. K.
                  Lau or C. C. Parry & E. Palmer.  A cursory attempt to
                  make all abbreviations unique has been started. 
                  Collectors with a common surname, even though their
                  first and middle names are different, may have the
                  same first and middle initials.  For example, John
                  Michael Jones (1767-1840) and James Madison Jones
                  (1925-x) might both be abbreviated J. M. Jones.  If
                  they have been given unique abbreviations, the older
                  will be abbreviated J. M. Jones and the younger will
                  appear as James M. Jones, but this has not been done
                  for all entries.
4.  Reference    :  the source of the information for this collector.
5.  Country      :  the country or countries where this person(s)
                  collected.
6.  Group        :  particular groups of specialization, for example,
                  algae mosses, Asteraceae.
7.  Alt. Name    :  includes alternative names this collector may have
                  used, such as married or maiden names, or earlier or
                  later names.
8.  Comments     :  includes more specific collecting localities,
                  herbaria where specimens are deposited, numbers of
                  collections, etc.
9.  Record Number:  the number from HUH's database system.
10.  Last Edit   :  the date this record was last edited.

9:12 PM 5/30/96-tried to separate by $ and delimit with nothing - no help - this file has weird structure - tried the expert with $ as separator but no delimiter - it worked, but special chars garbled - will try again

9:38 PM 5/30/96-fewer garbles if 'dos' selected for type of text BUT now field lengths clearly not good - trying again - worked pretty well, but now have fields that are too long - will either fix in pdox or do it in kman

7:14 AM 5/31/96-check this am reveals only 9 fields - did not take edit date - will try again although really don't need this

7:39 AM 5/31/96-have to define huhref# as a text field and give it a length - this done, it works for all 10 fields - now grossly over length and not sure how to deal with this - only problem now is special chars - most ok, but have some:  Bj"rlykke, Bj"rn and will have to deal with this somehow

7:56 AM 5/31/96-tried to 'repair', but rebuilding process only created a backup copy - sort stopped - not enought disk space - time to clean up - set up FLORA/collectors and shipped:

harvcol1.db - 63,890kb - newly formed pdox table
harcol1.doc - 7,428 - word doc from un-zipped base file
harcol1.txt - 6,839 - text output from word doc (used for import)
harvcoll.zip - original download from huh - 2124 Kb    Thu Apr 04 20:53:00 1996 (from their FTP site:   ftp://huh.harvard.edu/pub/authority_files/botany/collectors)
harvdoc.doc - word version of the readme file from HUH

will delete all but db from sheba and asked huh about field length - now dealing with endemics/voting and the FTC

9:38 AM 5/31/96-Natural Resources - Program Head - David Riskind - 512/389-4679 [edit ftc members]

2:28 PM 5/31/96-export of harvcol1.db to text and attempt to attach to kman table huhname1.itb complitcated by use of quotes in the detailfield and maybe other - so, will try to export as a dbf and see what happens

3:42 PM 5/31/96-export to dbf and attach worked fine - deleting dbf file and will not check string lengths on fields of kman table to truncate to proper size

5:03 PM 5/31/96-checks show some very long strings - very few - weird table

9:18 AM 6/1/96-checked string lengths and allways a few records with long strings - redefined, but did not trim the table much.  sorted by staname, added 4 char field - plucker - and set up huhadd1.ipf with huhstat.htm to add this pluck string and track reps - lots of reps and can check by stating huhstat - now will mod tracyin3 portions that deal with pluck, to array, selection from array to set up a routine - to be inserted in or called from tracyin3 - that will get a collector name

12:07 PM 6/1/96-extract from tracyin3 for collector name grabbing = tracoll1.ipf - working roughly now

11:15 AM 6/2/96-to add new names to huhname1, setting collrefs.itb to maintain updated reference numbers and will use 'attach1' to update both huhname1 and collrefs

2:06 PM 6/3/96-still working on tracoll1.ipf, went down to WOB with susan and mr - got a map for imaging - spend ca. 1 hr on revising aspt to a frames mode using the rubber band ball page as a model

7:49 AM 6/4/96-eddy has some nice jpgs of WOB, new, frame-based aspt pages look nice, have tracoll1.ipf managing name strings, now to attach to both huhname1 and taesdata (with new fields)

1:23 PM 6/5/96-writing TRACOLL3.IPF from TRACOLL2.IPF [FUNCTIONAL STAND ALONE] for connection to TRACYIN3.IPF which will be renamed to next version for conneciton to this - and taesdata will be expaned to include a 100 char name field, a 12 char huhref field, and date/col# fields

4:34 PM 6/5/96-all done, now will set up tracyin4 and try to get all variables squared away between the two programs

9:27 AM 6/6/96-sheba screen - fairly stable for the past week, is starting to fail regularly - will look at transfer of all to the csdl for subsequent transfer to the p100 and elsewhere - have the two programs working together and a check form for the final record, now fixing color shifts and trying to figure how to delete a record if the inputter has a problem with non-editable items in the check form

11:00AM  6/6/96 - screen pretty well shot on sheba - all files now on p100 and now editing this file from there

12:49PM  6/6/96-tried to shunt as much as possible from kman dirs on sheba - put fws stuff in 'endangered' and transferred all aspt and nm-grasses ipf, itb, ind files

1:52PM  6/6/96-only 40mb+ availble on trixie - will have to get another computer to run the tracyin4/tracoll3 thing - will check jrm's 486

2:54PM  6/6/96-jrm's machine has 2 drives, d is the biggest and newest - has 70 mb free BUT no enet card?!!

4:58PM  6/6/96-think most of the bugs are out, but still hacking - should try straight run tomrrow and look for record check after collector name addition

2:24PM  6/7/96-now revisiting the arrary error in the county add - some probem with row# vs. name# that am hacking out now - also, two brazos county records got into tracount and don't know how that happened

3:53PM  6/7/96-problem was with no resetting WITHIN the county loop - fixed and all seems to be ok now

6:56AM  6/8/96-lots of testing on tracyin4 and all seems to be ok - brought the 486 from home - no ethernet - need to switch card from trixie or maybe get a new one installed - now need to start taking are hard look at:

1.  setting up something that will allow translation of herb1.itb to 'phase I' condition with full BONAP fields, col names, etc.

2.  Update my home page


3.  set up something for bonap file for menu/open mg search
xxxxxxx
8:33AM  6/8/96-so, looking at herb1 - have:
1st 4 of family name
1st 5 of genus namd
full species epithet
1st initial and 1st 4 of collector name with collector# and month/year of collection

need:

1.  confirmation of full name with infra with grab of all bonap name fields, codes, and author once onfirmed

2.  entry of collection day

3.  full collector name

Approach will be to use herb1.itb as a base - entry will be via herb1.itb and will probably have to add a tag string to this to mark records that have been 'translated' to tamudata.itb.  the new table has to be structured as taesdata.itb.  thus, collector name will have to be added to huhname1 with a 'tamu+' huhref# string and will have to set up two county id strings - fullname and utcode - can use tracyin4 as a base, but will have to use full bonap table for names BUT should start with the grasses