Hari Kiran's Blog
How to Save Data to MongoDB from a CSV file using Mongoose in Node.Js

How to Save Data to MongoDB from a CSV file using Mongoose in Node.Js

A Beginners Guide

Hari kiran Kante's photo
Hari kiran Kante
·

8 min read

Table of contents

Introduction

In our data-driven world, the ability to efficiently store and manage information is a critical aspect of application development. MongoDB, a NoSQL database, and Node.js, a versatile JavaScript runtime, offer a dynamic duo for handling and persisting data. And CSV is a very simple and flexible way to store data. So, it’s not a surprise if you encounter a situation where your data is in a CSV file that you want to store in a Database for use in the application. If you are someone with the same requirement and you are struggling with how we can do this by writing some kind of script, then this blog is for you.

The actual reason for writing this blog is because, I have seen a friend of mine dealing with a similar situation So, I thought If I write a blog, it might help other beginners like her.

We'll begin by setting up a Node.js project, ensuring you have the right prerequisites in place. From parsing CSV data to creating a MongoDB schema, connecting to the database, and saving your data, this comprehensive guide covers every facet of the journey.

Prerequisites

Before starting Let’s make sure that we are on the same page:

  • I expect that you already have some experience with Java Script and have Node Js installed on your system

  • You can use either a local Mongodb or a database from Mongo Atlas whichever you are comfortable with.

  • With that we are ready to go, open your favorite text editor and get ready to write some code.

MongoDB and Mongoose Library

As I mentioned, we are going to use mongoose library to store data in Mongo DB. Let me give you a brief about both of them. Mongo DB is a popular NoSQL database which is also known as document-oriented databases. Just like we have a table in SQL, we have collections in No SQL and just like we have rows in each table in SQL, we have documents in each collection in No SQL. The flexibility with No SQL databases is that we don’t have to follow any rigid schema for documents.

Mongoose is a data modeling for Mongo DB in Node JS. It provides a schema-based abstraction for working with MongoDB. It allows you to perform database operations, and data validations more easily. In mongoose, we work with models that allow us to perform all crud operations on the database. These models are created by schemas that represent the structure of the data.

Preparing the Data

Let’s start by creating our csv data. If you have your own csv data then, it is totally fine else we can generate some dummy data online. Use this website Mockaroo to generate some data in csv format to follow along.

My CSV data looks like below

Setting Up Node.Js

To setup the nodejs project, create a new folder and open it in your favorite text editor. Now, open the terminal and run the below command to start the project setup

npm init -y

It will create a package.json which is necessary for node js to manage packages. Now install the necessary packages like csv-parser and mongoose using below command

npm install csv-parser mongoose

csv-parser is used to read the data from the csv format and mongoose is for connecting to the mongodb and do crud operations. Now, create a index.js file where we are going to write our main code.

After doing above changes, this how your folder structure should look like

Creating Schema and Model in Mongoose

Now, our environment and data is ready, let’s focus on create the models which will help us in saving the data into database. In order to create a model we need schema and that’s where we will start. You can create this schema in the same file or your create another file for this and create it there. I am creating another file with the name productModel.js.

In that file, start by importing the mongoose module

const mongoose = require("mongoose");

Now, create a schema representing your data. Here my data is related to products which has following field like product_name, price, quantity, and description. So, I am going to create a schema representing my data like below

const productSchema = mongoose.Schema({
  product_name: String,
  price: Number,
  quantity: Number,
  description: String,
});

As we can see we are creating the schema using the Schema class from the mongoose module. We have to pass a object where the keys are the fields of our data and values are their data types. There is so much that you can do with schema. But, that’s not our focus right now.

Now, create a model from the schema as shown below

const product = mongoose.model("Product", productSchema);

Again, we are using a function called model from the mongoose module. The first argument is the name of the model and second argument is the schema that we created earlier. This function will return an instance of Model class.

Now, If we want use this model in other files, we need export it.

module.exports = product;

With that, we can import this model in other files. Now, your productModel.js should look like below

const mongoose = require("mongoose");

const productSchema = mongoose.Schema({
  product_name: String,
  price: Number,
  quantity: Number,
  description: String,
});

const product = mongoose.model("Product", productSchema);

module.exports = product;

Reading Data from CSV File

Let’s come back to index.js file and here we are going to use the csv-parser to read the csv data from the file. Start by importing the fs module which will help in read the file and csv-parser which help in parsing the content read from the file, into js objects.

const fs = require("fs");
const csvParser = require("csv-parser");

We can read the file using the createReadStream function in the fs module.

fs.createReadStream("./data.csv")

we have to provide the path to the file as an argument to the createReadStream function. It will create a stream object which has a method called pipe. pipe method is just like pipe | symbol in linux where we pass output of one command as input to the other command.

fs.createReadStream("./data.csv")
    .pipe(csvParser())

Here, this method also takes the stream and passed it to the csvParser. Now every time the csvParser parses a line it will generate an event called "data". We can pass a callback function to this event and do what ever we want which that data.

fs.createReadStream("./data.csv")
    .pipe(csvParser())
    .on("data", (row) => {
    console.log(row);
  })

Now if you run the file using the command below in your terminal.

node index.js

You can that the data has been converted into js objects. Here is the output

The callback function is called for each row in the csv file. Now we are able to parse the data present in the csv file. Let’s connect to the database and save this information there.

Connecting to Database

We will use the connect method from the mongoose module to connect to the database in the index.js file. Like below

const mongoose = require("mongoose");
mongoose.connect("mongodb://127.0.0.1:27017/testdb");

It will connect to the local mongodb you can also replace it with the mongo atlas url.

Saving Data into Database

It’s time to import the product model that we created earlier and use it to save the data into database.

const Product = require("./productModel");

Now, we can use this model to create a new object of the product and save them to the database. So, we are going to write this logic in the callback function which will be called every time a row is read from the csv file.

fs.createReadStream("./data.csv")
    .pipe(csvParser())
    .on("data", (row) => {
      const product = new Product(row);
      product.save();
    })

So, every time a new row is read, a new Product object will be created. Here Product is nothing but the Model and product is instance of that Product model. The model object have a method called save which will save that object to the database.

Now add some try…catch block to handle the errors. I added one more on method to print the line “Data has been successfully inserted’ onto the console at end of inserting all the rows. After doing the above changes your code in the index.js file should look like below

const fs = require("fs");
const csvParser = require("csv-parser");
const mongoose = require("mongoose");
const Product = require("./productModel");

mongoose.connect("mongodb://127.0.0.1:27017/testdb").catch((err) => {
  console.log("Error", err.message);
});

try {
  fs.createReadStream("./data.csv")
    .pipe(csvParser())
    .on("data", (row) => {
      const product = new Product(row);
      product.save();
    })
    .on("end", () => {
      console.log("Data has been inserted successfully.");
    });
} catch (error) {
  console.log(error.message);
}

Now, run the above file using the command

node index.js

And check your database.

Just like that with just few line of code, One thousand documents have been inserted into the database.

Conclusion

In this blog, we've embarked on a journey to make data storage a breeze. By connecting the dots between CSV files, MongoDB, Mongoose, and Node.js, we've simplified a crucial aspect of web development.

We have created schema and model with mongoose library and then we are able to parse a csv file using a csv-parser library. Then, we connected out script to mongodb and used the model to save the data into the database.

Remember, this is just the start. The world of data manipulation is vast, and your skills will only grow from here. Explore advanced concepts, refine your error-handling techniques.

Thanks for Reading

I hope this guide has demystified the process of saving data from CSV to MongoDB, empowering you to take control of your data with confidence.

If you have come this far, I appreciate your commitment towards learning. If you have questions, feedback, or topics you'd like us to cover in future blogs, please feel free to reach out.

I would like give the credit of this blog to my friend Ishika who has given the idea for this blog.

Your coding journey never stops. Keep coding 🚀📊

Bye People! 🙋‍♂️

Node.jsmongooseMongoDBcsvbeginner