CONTACTS 

SUBJECTS
Converted Data Set on Internet to Revised HTML Format as of 021225 Us
DOSE Commands Failing to Execute DOS Commands
DOSE Failing to Execute Commands from Within Medit

1705 -    ..
1706 - Summary/Objective
1707 -
170701 - Follow up ref SDS 25 0001, ref SDS 22 0000.
170702 -
170703 - Gary reported difficulties with the 0126 processor created a week or
170704 - so ago for calling 070405 to convert SDS records on the Internet to
170705 - the current improved version that is more HTML compliant, ref SDS 0
170706 - TE4G for display on Gary's browser.  Since the main problem is memory,
170707 - considered a revision to process each file separately in a dose
170708 - session. ref SDS 0 TFGF  This new routine is launched from the Tools
170709 - menu. ref SDS 0 K44O  We are only processing the directory for my SDS
170710 - records, since Gary's have been produced with the updated code in
170711 - 070405.  Had several successful runs.  Final run processed 6K files in
170712 - about 33 minutes for the entire 00101 directory. ref SDS 0 TF6M  After
170713 - the transfer directory has the complement of 6K records ready for
170714 - uploading, couldn't process any more records individually, because
170715 - Medit cannot maintain that many files at once, so had to go ahead and
170716 - remove files from Internet and then uploaded revised files, to clear
170717 - the transfer directory. ref SDS 0 H83I  This was completed the next
170718 - day OA 1100.
170719 -
170720 -      [On 021226 notified Gary records on Internet have been converted.
170721 -      ref SDS 26 0001
170722 -
170723 -
170725 -  ..
1708 -
1709 -
1710 - Progress
1711 -
171101 - Conversion of Existing HTML on Internet
171102 -
171103 - Received ref DRT 1 0001 from Gary dated 021223, following up on the
171104 - report submitted 021219, ref SDS 25 0001, and explaining today that...
171105 -
171106 -   1.  I broke the files up into lists of 20. Conversion generated 2776
171107 -       HTML files with differing number of lines.
171109 -        ..
171110 -   2.  I have attached the file lists of files and the batch file which
171111 -       executes them. Each list requires pressing <escape> when the
171112 -       list finishes since macro 0126 doesn't quit Medit.
171113 -
171114 -   3.  I am putting this effort on hold for a while. I have some other
171115 -       projects that I simply *have* to put some effort into.
171116 -
171117 -   4.  My next steps on SDS involve laying out a modularization
171118 -       approach and testing the idea with a compiled version of my
171119 -       script to clean up the WT transfer directories.
171121 -  ..
171122 - Sent Gary an email linked to this record, letting him know I will work
171123 - on this.  Objectives are reported on 021206. ref SDS 23 0001
171124 -
171126 -  ..
1712 -
1713 -
1714 - Analysis
1715 -
171501 - Maybe we can do this without breaking the bank on memory.
171502 -
171503 -    1.  Get a list using sf based on the year.  We have 10 years or so,
171504 -        that is however many lists, each with less than 1,000 records,
171505 -        since we only have about 6K files to process, and these can be
171506 -        read in separately to do the whole thing in one process.
171508 -         ..
171509 -    2.  Read the filename and instead of opening the file run a dose
171510 -        command to process each file at the DOS prompt with a fresh
171511 -        set of memory.
171512 -
171513 -        I think the problem Gary reports, per above, ref SDS 0 TE4G, is
171514 -        that the process requires opening the actual SDS records in
171515 -        order to call 070405, and this fragments RAM.
171517 -  ..
171518 - This alternate method may be more stable.
171519 -
171520 -      [...below, after a lot of fixing up, got it to run to completion
171521 -      in about 33 minutes for 6100 files. ref SDS 0 TF6M
171522 -
171523 - Create a new routine...
171524 -
171525 -
171526 -                    c: sd 03 0127
171527 -
171528 -
171529 - ...which will work differently from 0126 developed on 021213.
171530 - ref SDS 24 TS5M
171531 -
171532 -
171534 -  ..
171535 - 0127 Convert All SDS Records in WL to WT for Updating HTML on Internet
171536 -
171537 -
171538 - Line 30, ref OF 1 HE5G, top of 0127
171539 -
171540 -    dose dir c:\sd\wl\sd\08\ > jyz
171541 -    e jyz
171542 -
171543 -       Get a list of users from directory at username level of SDS
171544 -       records in the local set of SDS records stored on the Internet,
171545 -       and put the list in a file jyz
171547 -        ..
171548 -       Open the file jyz.
171549 -
171550 -       For some reason this is not working, so we actually jump this
171551 -       beginning code.
171553 -        ..
171554 -       Curiously, all of the other dose commands execute correctly.
171555 -
171556 -         [On 021220 called Morris; he will investigate. ref SDS 27 0001
171558 -  ..
171559 - Line 60, ref OF 1 OZ7K, -label jmp1
171560 -
171561 -    setcnt 156 88
171562 -    setcnt 157 @156
171563 -    setgbl 156 @156
171564 -
171565 -       Since we cannot the initil dose command to work, per above,
171566 -       ref SDS 0 7P5H, for getting a list of users, will plub this to
171567 -       00101 and start with the year 1988.
171568 -
171569 -       Create a batch file to get a series of files that list SDS
171570 -       records for each year, beginning with 1988.
171571 -
171573 -  ..
171574 - Line 90, ref OF 1 0K4K,
171575 -
171576 -    datecnt 3 0
171577 -    setcnt 4 @3
171578 -    getgbl 1 156
171579 -    -if @1 > 87 addcnt 1 1900
171580 -    -if @1 < 88 addcnt 1 2000
171581 -    subcnt 3 @1
171582 -
171583 -       Use datecnt to determine the end date of the process, and
171584 -       subtract to get the value for the number to be processed.
171586 -  ..
171587 - Line 90, ref OF 1 LH4I, -label jmp1 in 127 about 40 lines below
171588 -
171589 -    e tst4.bat
171590 -
171591 -       Batch file will set up temp dir for storing the stuff, so it is
171592 -       easy to clean up at the end, per below. ref SDS 0 QE7J
171593 -
171594 -    macro 301
171595 -
171596 -       Empty file and leave a line to start the work.
171598 -     ..
171599 -    ins_text!md \rhfn!
171600 -    immed 2a
171601 -    up
171602 -    ins_text!cd\rhfn!
171603 -
171604 -       Create a directory and change to that directory to do the
171605 -       work.
171607 -  ..
171608 - Line 120, ref OF 1 RL6J, -label lpa in 0127
171609 -
171610 -    -label lpa
171611 -    *t88
171612 -    ins_text!c:\sd\01\sf c:\sd\wl\sd\08\00101\02\88\ *.* > 88!
171613 -    up
171614 -    -if @0 > 0 -goto dlpa
171615 -
171616 -       Build a batch file to get the list of files for each year, since
171617 -       Medit cannot open the whole thing all at once, per above.
171618 -       ref SDS 0 TFGF
171619 -
171621 -  ..
171622 - Line 120, ref OF 1 SK6M, -label lp2 in 01027
171623 -
171624 -    dose /c c:\sd\10\tst4
171625 -
171626 -       Run batch file that generates the list of SDS records in each
171627 -       year directory for my user ID in sd wl.
171629 -  ..
171630 - Line 200, ref OF 1 L95N, -label lp2a in 0127 about 60 lines below
171631 -
171632 -    Test code with debug here.
171633 -
171635 -  ..
171636 - Line 270, ref OF 1 OF8J, -label elp2 in 0127
171637 -
171638 -    e tst4.bat
171639 -    macro 301
171640 -
171641 -       The batch file is held in memory to the end, and then modified
171642 -       with new contents for cleaning up the temporary work files.
171643 -
171644 -    ins_text !del *!
171645 -
171646 -       Delete all temporary work files.
171648 -     ..
171649 -    immed 3a
171650 -    up
171651 -    ins_text !cd\sd\10!
171652 -
171653 -       Change directory back to default.
171655 -     ..
171656 -    up
171657 -    ins_text !rd \rhfn!
171658 -    up
171659 -    ins_text !del c:\sd\10\tst4.bat!
171660 -
171661 -       Remove the temp directory and delete batch file.
171662 -
171664 -  ..
171665 - Line 290, ref OF 1 HK9O, -label elp2 in 0127 about 30 lines below
171666 -
171667 -    save c:\sd\10\tst4.bat
171668 -
171669 -       Now that the batch file is built save it to default temp
171670 -       directory.
171671 -
171672 -    ecur 2
171673 -    dose /c c:\sd\10\tst4.bat
171674 -
171675 -       Return to original file and run batch op to remove temporary
171676 -       work files.
171678 -     ..
171679 -    purge tst4.bat
171680 -
171681 -       Remove batch file from memory.
171682 -
171684 -  ..
171685 - Line 40, ref OF 5 6J5O, -label s06116 in 070405
171686 -
171687 -    This is where code is processed using the actual filename.
171688 -
171689 -
171691 -  ..
171692 - Confidentiality Prevent Inadvertant Publishing Private Records
171693 -
171694 -
171695 - Line 90, ref OF 5 1889, -label svRcd in 070405
171696 -
171697 -    Added code to quit on confidential records, so they are not
171698 -    published to the Internet, using similar code for when a blank file
171699 -    is encountered, ref OF 5 9P7G, which occurs for reasons explained
171700 -    below. ref SDS 0 TG5L  This accomplishes work plan requirements
171701 -    confidentiality reported on 021206. ref SDS 23 G53J
171702 -
171703 -
171705 -  ..
171706 - Menu Operation Makes Calling Data Base Exchange Fast and Easy
171707 -
171708 - Decided to put this in following menus....
171709 -
171710 -       "Tools" at the bottom in Medit, ref OF 2 Q24H
171711 -
171712 -       "Tools" at the bottom in SDS records, ref OF 3 Q24H
171713 -
171714 - This makes conversion op easy to call when needed to transfer another
171715 - set.
171716 -
171717 - It seems likely this utility can be used, perhaps with modifications,
171718 - for a variety of mass data base operations.
171719 -
171720 - Essentially, this lets someone maintain and customize the set on the
171721 - local server, and then conform the set on the Internet.
171722 -
171723 -
171725 -  ..
171726 - Testing Eventually Demonstrated Can Process Entire Data Base
171727 -
171728 - Initial testing required a lot of revisions, and some further editing
171729 - on the record set in...
171730 -
171731 -                      c: sd wl sd 00101
171732 -
171733 -
171734 - ...as shown below. ref SDS 0 TE7I  First trial runs only produced
171735 - about 4K files, or 2K short.  So, was concerned.
171737 -  ..
171738 - Finally got the thing to run successfully without any errors.
171739 -
171740 -
171742 -  ..
1718 -
1719 -
1720 - 0241
1721 -
172101 - Start test.
172102 -
172104 -  ..
1722 -
1723 -
1724 - 0317
1725 -
172501 - Complete run without any problems in about 36 minutes.
172502 -
172503 -   Processed in wl   6,162 files  3,266 folders 68 MB  79 MB on disk
172504 -
172505 -   Produces in wt    6,044 files  3,193 folders 66 MB  77 MB on disk
172506 -
172507 -      There are two categories that reconcile some of this
172508 -      difference...
172509 -
172510 -         1.  Confidential records are not processed for transfer to the
172511 -             Internet, per above. ref SDS 0 ON5I
172512 -
172513 -             Theoretically, they should not be in wl, but they could
172514 -             have been converted prior to developing code that
172515 -             prevents conversion, and some may have been converted for
172516 -             display locally, and deleted from wt prior to transfer.
172518 -              ..
172519 -         2.  Records deleted after having been converted at a time when
172520 -             they existed.
172521 -
172522 -
172523 -
172524 -
1726 -

SUBJECTS
Upload Records (FTP)
Upload Converted Records Using Macro 127 that Conforms to Content of
Delete Data From Web Server

200502 -  ..
2006 -
2007 -
2008 - 0450
2009 -
200901 - Upload Converted SDS Records to Welchco on Internet
200902 -
200903 - After getting the list of converted files in sd wt, could no longer
200904 - convert SDS records and external documents, because the first step is
200905 - to maintain the transfer directory, and since it now has 6K files,
200906 - this is too big for Medit to handle.
200907 -
200908 - So started an op to upload the directory....
200909 -
200910 -                c: sd wl sd 08 00101 *.*
200911 -
200912 - First step is deleting the existing data base.
200913 -
200914 -     This seems to take quite awhile because a separate command seems
200915 -     to be sent for each directory, and we have 3K of these for the SDS
200916 -     records in my directory. ref SDS 0 TF6M
200917 -
200918 - Seems like it might take 3 or 4 hours to upload, even with DSL.
200919 -
200921 -  ..
2010 -
2011 -
2012 - 0558
2013 -
201301 - Completed deleting existing data set from the Internet.
201302 -
201303 - Began to upload.
201304 -
201305 - This may take 4 hours, so will get some sleep.
201306 -
201308 -  ..
2014 -
2015 -
2016 - 1118
2017 -
201701 - Completed replacing all SDS records for 00101 directory.  Took about 5
201702 - hours including several delays when the DSL line crashed, and had to
201703 - maintain transferred files manually.
201704 -
201705 -
201707 -  ..
201708 - Some Files Crashed During Processing and Were Fixed or Deleted
201709 -
201710 -
201711 -        880930 Clyde called............ ref SDS 1 0001
201712 -        890808 WMA marketing.....
201713 -
201714 -           Has been deleted on d: so deleted on c:
201716 -         ..
201717 -        890816 WMA marketing.....
201718 -
201719 -           Has been deleted on d: so deleted on c:
201720 -
201721 -        890918 Called Ross re....
201722 -
201723 -           Has been deleted on d: so deleted on c:
201725 -         ..
201726 -        890929 WMA marketing.....
201727 -
201728 -           Has been deleted on d: so deleted on c:
201729 -
201730 -        891227 Call Ross............... ref SDS 2 315N
201731 -
201732 -           Do not need this on the Internet, so deleted it
201733 -
201734 -        891227 Call Norm and Jeff...... ref SDS 3 0001
201735 -
201736 -           Do not need this on the Internet, so deleted it
201738 -         ..
201739 -        900329 WMA marketing call......
201740 -
201741 -           Do not need record, so deleted it.
201742 -
201743 -        910221 Dale Ames here to....... ref SDS 4 0001
201744 -
201745 -           Confidential record, should kick out.
201746 -
201747 -        910420 Meeting with Morris..... ref SDS 5 0001
201748 -
201749 -           Fixed this record.
201750 -
201751 -        920127 Evaluation of SDS.......
201752 -
201753 -           Record by another user was published under 00101
201754 -           fixed this.
201756 -         ..
201757 -        940215 Meeting with Port of SF.... ref SDS 7 0001
201758 -
201759 -           Fixed this record, it was entered under my User ID, but is
201760 -           for Kwan Henmi.
201761 -
201762 -         940327 Meeting Sacket design..... ref SDS 8 0001
201763 -
201764 -           This should kickout because it is confidential
201765 -
201766 -         970910 Received notice to PMI....
201767 -
201768 -           Has been deleted on d: so deleted on c:
201770 -          ..
201771 -         990224 Meeting with Major Blechinger...
201772 -
201773 -           Has been deleted on d: so deleted on c:
201774 -
201775 -
201776 -
201777 -
201778 -
201779 -
201780 -
201781 -