This is a small project I made to parse csv or pipe delimited files into JSON while cleaning and refactoring code from previous projects.
I had previously made a small text engine to process csv, xml, json, or pipe files to a SQL database but I was curious how challenging it
would be to convert basic text files to JSON. To be clear, this can be done fairly easily by implementing packages but I wanted to try it
out for myself.
To process bulk csv and pipe files I created a parser that finds all files of type csv or txt in the target directory. It then processes
each of them into an equivalent JSON file, saving them to a new out folder in the target directory. Initially, I accomplished this via
splitting lines of the files by their delimiter. This was a very quick and efficient approach but it did run into problems when csv data
had commas within the values. I go over how I handled this edge case here.
Once the file had been processed into string arrays, a new file is created and each line is written in as a JSON object.
Default Read Process
Write Process
Multi Threading
After succesfully building out the core of the project, I wanted to look into ways to make the process faster. At that juncture, all files were being processed
sequentially, large files would slow the process down significantly and the speed just was not what I had hoped. To overcome this I decided to process each file
on a seperate thread. This was one of the first instances where I have worked with threads so I had to do a bit of research but I was easily able to speed up the
process through this one simple step. In the future, I want to focus on learning more about best practices and strategies for more complex multi-threading.
Multi Threading
Edge Cases
One of the issues that I was aware of from the start was the issues generated by csv values that contain commas. I was not sure how I would approach this at
first so I opted to create a general approach before figuring out how to implement checking or workarounds for this edge case.
When it came time to find a solution, my research led me to the visual basic text parser. I was somewhat thrown off by the inclusion of visual basic usings
but was easily able to implement the text parser via the same Idisposable using statements.
