Dave Kurman / Computer Systems Analyst / Expertise in computer programming, database, web development, system administration

© 2024. All rights reserved.

BFG Repo-Cleaner

30 May 2017

BFG Repo-Cleaner is a good tool to clean up a Git repository before sharing

Removes large or troublesome blobs like git-filter-branch does, but faster. And written in Scala.


bfg --strip-blobs-bigger-than 100M --replace-text banned.txt repo.git

Replace all passwords and listed in a file (prefix lines ‘regex:’; or ‘glob:’; if required) with ***REMOVED*** wherever they occur in your repository :


bfg --replace-text passwords.txt  my-repo.git

Delete all files named ‘id_rsa’ or id_dsa’ :


bfg --delete-files id_{dsa,rsa} my-repo.git

Using BFG on Windows


java -jar bfg.jar $*
  • Make a replace.txt file:

PASSWORD        # Replaces PASSWORD with ***REMOVED***
PASSWORD==>TEST # Replaces PASSWORD with TEST
PASSWORD==>     # Replaces PASSWORD with blank
                # You can use Regular Expressions also
  • Run this command in the repo folder from Git Bash or Win10 Bash:

bfg --replace-text replace.txt
  • When bfg is done run these commands

git reflog expire -expire=now -all && git gc -prune=now -- aggressive

git push                   # -f may be needed

Notes

  • I had already pushed the repo and shared it.  That got complicated by multiple branches.  My fault.  It would have been much simpler to run bfg before sharing.
  • I fumbled a bit trying to get bfg running in Windows command shell.

Resources

Microsoft Windows File Server Locked File Watcher

29 May 2017
  • It is common for companies to work with Excel, Word and other files on Windows network shared folders.
  • More than one user cannot usually work on files like Excel at the same time.
  • When trying to open a locked network file it can be hard to see who has it locked.
  • The best method to see who has a file locked is from the Windows file server. But that requires Administrator level access. That level of access is not practical for all users for security reasons.
  • This tool has two parts:
  • A Powershell process that watches for locked files and saves the result.
  • A web server that displays the locked files and who has them locked.
  • No special access is required to view the web results.The Powershell process uses
  • The Powershell process uses a list of watched files and folders. This limits the number of files displayed. This also allows it to avoid showing sensitive files or network folders.

Visit Github to download.

Cliplink -- a utility for short, reliable links on Windows networks

06 Apr 2017
  • Users wanted a way to create reliable links to files and folders on Windows network drives.
  • Spaces and special characters break Outlook’s hyperlinks to network files. Clipboard-Link fixes that problem.

How do I get set up

  • Get Cliplink from => GitHub.
  • There is no configuration
  • There are no dependencies
  • No database configuration
  • Save the contents of this repo to a network drive or share.
  • Tell users to run setup from there.

How does it work

  • Locate the file or folder on the network.
  • Right-click and select Send To > clipboard-link
  • Cliplink creates a safe link to the file and puts it on the Clipboard.
  • Paste the link into Email, documents, etc.
  • Create a link to a network file or folder with Clipboard-link and paste into a message
  • Select the link part and Cut
  • Select the name part between the brackets
  • Select Insert menu > Hyperlink
  • Paste the link in the URL box
  • Click Ok
  • Clipboard-link copies the file/folder name and shortened link to the clipboard

  [Engineering Documentation]                  file://F:\ENGINE~1

          ^ Name ^                               ^   link  ^

NOTE: **Cliplink will not create links for executables [files ending with EXE, COM, BAT, etc.].

octocat

Confluence, a Great Documentation Tool

06 Apr 2017

Introduction

Years ago, I was the new system administrator at the company. I had to get a handle on the multitude of systems, servers, and software. My predecessors did a great job of documenting the details in Microsoft Word. Because of the number of documents, it was hard to find stuff. It was time-consuming to keep details up to date as software and systems changed.

Then I discovered MediaWiki, the Open Source software that runs Wikipedia. An old donated PC was the host for Linux and MediaWiki. I copied the Word documents into the wiki. It was easy and natural (for me at least) to build easy to navigate links to the IT documentation. MediaWiki keeps older versions so it is easy to see the history of changes. Wiki markup is weird and unnatural to normal people but a few folks at the company took to it. The help was welcome for maintaining the IT processes and procedures. Contributors logged in to MediaWiki so we could see who made changes.

Unlike the Word documents, it was easy to search and find information. We could email hyperlinks to folks in need of support. That saved time for me and other power users. And normal folks would sometimes let us know when documentation was out of date or wrong. MediaWiki reduced the friction of sharing and managing the documentation.

I looked at SharePoint a few times. SharePoint is well integrated with Windows and Microsoft Office products. It seems like a natural extension for folks familiar with that world. SharePoint’s wiki didn’t work for me though. And the easy extensibility worked against using it for documentation. That lead to a lot of rabbit holes to get lost in when the focus should be elsewhere. Don’t get me wrong, I am very good in the Microsoft world but it felt like more friction to get things done.

Atlassian Confluence

I came across Atlassian Confluence while exploring Agile development methods. Confluence is like a polished MediaWiki.

Working together on documentation is well thought out. Instead of emailing the author about changes a user selects text on the Confluence page. A popup presents the user with the option to comment right on the page. Comments are part of the documentation. That is much better than email silos.

The real-time collaboration is great. When more than one user edits a page at the same time the changes synchronize between them in real time.

Like MediaWiki, Confluence keeps track of previous versions of the pages. Having that history gives users more confidence. They do not have to maintain many copies of a document. The history is all right there with that page in Confluence.

Embedding and attaching pictures is easy in Confluence. Users can also attach Word, Excel, PDFs and other files.

A cool plugin for Confluence is Gliffy. It is like a web based Visio for creating diagrams, flowcharts and so forth. Many times IT documentation is better with a network diagram or a business workflow. The Gliffy plugin tracks versions as well. Users have that history of changes which helps with complex diagrams.

Confluence benefits from tight integration with Atlassian JIRA. That is good if you use JIRA. JIRA is an issue tracking tool that is great for software development. Many use JIRA for business project management and managing service.

Confluence and JIRA are both available as cloud services from Atlassian. No servers to maintain.

Having time on my hands I went with installing Confluence and JIRA on my servers. Well, actually Hyper-V virtual machines running Linux. The install process is well documented. I ran into a few problems but Atlassian had intuitive help to get through them. The price made me smile. My self-hosted servers were $10 each for 10 users. That’s a good value and a low barrier for these tools.

Confluence helps my efficiency with the DRY principle (Don’t Repeat Yourself). Having the notes and documentation online means they’re available almost everywhere. And Confluence reduces the friction of maintaining documentation.

CAD Drawings to PDF with SolidWorks, eDrawings and VBA

06 Apr 2017
An automated solution to make Acrobat PDF files from CAD drawings. The solution handles SolidWorks and AutoCAD drawings.
 

The problem

 
The company had a solution for plotting engineering drawings.  Employees in production had a fast utility to lookup CAD drawings. There were tens of thousands of drawings.  Printing was a simple button click.  There were no printer dialogs.  No printer choices.  No choosing settings.  Less wasted paper.  It was a huge time and resource saver.  With the upgrade to Windows 7 the supporting engineering tools no longer worked.  The efficient CAD printing no longer worked after the I.T. upgrades.
 

The parts

 
eDrawings is a free utility from SolidWorks.  eDrawings opens SolidWorks and AutoCAD drawing files.  eDrawings updates every year.  eDrawings handles new AutoCAD drawing formats with the updates.  Like most of SolidWorks, you can automate eDrawings.
 
novaPDF is a utility to generate Adobe Acrobat [.PDF] files.  novaPDF is available on Windows computers.
 
Microsoft Access 64bit.  Most use the 32bit versions of MS Access and MS Office.  For this solution we need a 64bit VBA capable container to host the eDrawings component.
 

The solution

 
With novaPDF create some virtual printers.  Create separate printers for each CAD drawing page size and orientation.  For example: ‘B_Size_portrait’, ‘B_Size_landscape’, ‘D_Size_landscape’, ‘C_Size_landscape’.
 
Create a blank MS Access database application.  Add a blank Form without a linked table.  Embed the eDrawings component into the form.  
 
Switch to the VBA editor for the MS Access Form.  The first step is a loop to watch for new SolidWorks or AutoCAD drawing files.  Opening each drawing file in turn.
 
Use the eDrawings component Event functions to handle the file once it opens.  The eDrawings component has properties to detect the drawing paper size and orientation.  Use those properties to choose the matching novaPDF virtual printer and print.
 
Provide error handling for the eDrawings Event functions for problems like corrupt files.  This app rejects undefined paper sizes.  We ran into a problem detecting AutoCAD drawing sizes that used different scales.  The solution was to calculate the size ratio of height to width and use the ratio to select the output paper size.  The MS Access/eDrawings solution becomes unstable after processing 40 to 50 files.  Auto-restarting MS Access after 20 to 30 files solves the problem.
 
This PDF Generator solution runs on a Windows virtual machine.  It averages processing drawing files every three to five seconds with moderate resources.  The solution keeps up with an engineering department of about a dozen.  Well over a hundred thousand legacy CAD files ran through the PDF Generator.
 
There are a few other pieces to the solution.  A lightweight, custom server process provides point and click printing.  A custom web server and front end JavaScript client provides searching and viewing.  The CAD library is now available to desktops, laptops, tablets and smartphones now. There are other, comercial solutions but this preserved the company’s workflow.