How to Generate Sitemap from a Dataset and Create JS Functions to Insert, Update, and Delete a URL

Nitish Kumar Singh

Dec 20, 2023

Hello developers! In this blog post, we will learn how to create an XML sitemap from a set of data and create JavaScript functions to dynamically insert a new URL, update, and delete an existing URL by running these functions on the server.

Photo by Domenico Loia on Unsplash

Let me describe the process of creating a sitemap from data sets. I have an array of my blog post data and want to generate a sitemap for all posts. So, the following JS function creates an XML sitemap from the posts array and saves it in the working (JS code running) directory. 

function createAndSaveSitemapFromData(posts,origin) {
    // Generating sitemap content from data and origin
    const xmlHeader = '<?xml version="1.0" encoding="UTF-8"?>';
    const urlsetStart = '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">';
    const urlsetEnd = '</urlset>';
    const urlTags = posts.map(post => {
        const url = origin + post.path;
        return `   <url>\n      <loc>${url}</loc>\n      <lastmod>${post.updatedAt}</lastmod>\n   </url>`;
    });
    const sitemapContent = `${xmlHeader}\n${urlsetStart}\n${urlTags.join('\n')}\n${urlsetEnd}`;
    // Save this sitemap content to sitemap.xml file in reuired directory by modifying path
    const fs = require('fs');  
    const relativePath = "/"; // modified it according to requirment
    const fileName = "sitemap.xml";
    const path = process.cwd()+relativePath+fileName;
    try {
        fs.writeFileSync(path, sitemapContent);
        console.log(`Sitemap saved at ${path}`);
    } catch (error) {
        console.error(`Error writing file: ${error.message}`);
    }
}

In the above code, we have added only the URL (required) and lastmod (optional) items in sitemap creation, but you can add more items as required in the return statement of the map function. We saved the sitemap file in the running code directory, and to save it in a different directory, modify the relativePath variable.

The following code shows an example of using the above function. 

const fetchPosts = ()=>{
    // Here fetch your actual data from which you want to build sitemap file, I return dummy posts
    return posts = [
        { path: '/post1', updatedAt: '2023-01-01' },
        { path: '/post2', updatedAt: '2023-02-01' },
    ];
}
createAndSaveSitemapFromData(fetchPosts(),'https://example.com');

Make sure the path and updatedAt keys are available in your provided data array items; otherwise, you will get an error. The above code helps you to generate a sitemap for an existing website.

Manipulating an Existing Sitemap

In my case, the idea is to create functions to dynamically update my sitemap on a deployed blogging website. So, whenever I post new blogs and update existing blogs, the sitemap file on the server is automatically updated by calling my created API to do this work from the post or update API.

But my idea does not work because my website is built with Next.js and hosted on Vercel, which serves sites using Serverless Functions that run on a read-only system but works fine on a traditional server.

To explore more about this problem on Vercel and to find a workaround, read my blog post How to Read and Write Files in Next.js on a Vercel Deployed Website.

To explore more about this problem on Vercel and to find a workaround, read my blog post "How to Read and Write Files in Next.js on a Vercel Deployed Website."

Insert a New URL in the Sitemap

Use the following function to insert a new URL by providing the URL and lastmod. 

const insertNewUrl = (url,lastmod)=>{
  const urlInfo = "<url><loc>" + url + "</loc><lastmod>" + lastmod + "</lastmod></url>";
  const relativePath = "/public", fileName = "sitemap.xml";
  let path = process.cwd() + relativePath + fileName;
  var data;
  
  const fs = require("fs");
  try {
    data = fs.readFileSync(path,"utf-8");
  } catch (error) {
    return {ok:false,message:"Error in reading to the sitemap file. Error: "+error.message};
  }
  let p = '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">';
  const insertPosition = data.indexOf(p) + p.length;
  const modifiedContent = data.slice(0, insertPosition) + urlInfo + data.slice(insertPosition);
  try {
    fs.writeFileSync(path, modifiedContent, "utf-8");
    return { ok: true, message: "New URL info succussfully inserted in sitemap file." };
  } catch (error) {
    return { ok: false, message: "Error in writing to the sitemap file. Error: " + error.message };
  }  
}

Run this function on the server to insert new URL info and change the relativePath as you want.

Update an Existing URL in the Sitemap

Use the following function to update the lastmod of an existing URL by providing the URL and lastmod. 

const updateLastmodThisUrl = (url,lastmod)=>{
  const relativePath = "/public", fileName = "sitemap.xml";
  let path = process.cwd() + relativePath + fileName;
  var data;

  const fs = require("fs");
  try {
    data = fs.readFileSync(path,"utf-8");
  } catch (error) {
    return {ok:false,message:"Error in reading to the sitemap file. Error: "+error.message};
  }
  const urlPosition = data.indexOf(url);
  if (urlPosition !== -1) {
    const lastModStart = data.indexOf('<lastmod>', urlPosition) + '<lastmod>'.length;
    const lastModEnd = data.indexOf('</lastmod>', lastModStart);
    const modifiedContent = data.slice(0, lastModStart) + lastmod + data.slice(lastModEnd);
    try {
      fs.writeFileSync(path, modifiedContent, "utf-8");
      return { ok: true, message: "URL's lastmod succussfully updated in sitemap file." };
    } catch (error) {
      return { ok: false, message: "Error in updating to the sitemap file. Error: " + error.message };
    }  
  }else return {ok:false,message:'Error in finding url'};
}

Delete a URL

Deleting a URL is rarely required, but we can use the following function to delete a URL from the sitemap. 

const deleteThisUrl = (url)=>{
  const relativePath = "/public", fileName = "sitemap.xml";
  let path = process.cwd() + relativePath + fileName;
  var data;

  const fs = require("fs");
  try {
    data = fs.readFileSync(path,"utf-8");
  } catch (error) {
    return {ok:false,message:"Error in reading to the sitemap file. Error: "+error.message};
  }
  const urlPosition = data.indexOf(url);
  if (urlPosition !== -1) {
    const lastModStart = data.lastIndexOf('<url>', urlPosition);
    const lastModEnd = data.indexOf('</url>', urlPosition)+"</url>".length;
    
    const modifiedContent = data.slice(0, lastModStart) + data.slice(lastModEnd);
    try {
      fs.writeFileSync(path, modifiedContent, "utf-8");
      return { ok: true, message: "URL succussfully deleted in sitemap file." };
    } catch (error) {
      return { ok: false, message: "Error in deleting URL from sitemap file. Error: " + error.message };
    }  
  }else return {ok:false,message:'Error in finding url'};
}

I hope you understand and learn how to create a sitemap from data sets and how to dynamically manipulate a sitemap file on the server. Happy coding!

Published on Dec 20, 2023
Comments (undefined)

Read More