Lots of small tasks today.

Site Edit Mode

A toggle button was added so that normally, the buttons to add and edit the navigation do not appear, as well as the drag handles. This meant that the API call for “Features” was not necessary. I found that the Redux Toolkit Query had a “Skip” option, in which I just flagged it as false until edit mode was toggled on. As the edit mode was toggled on/off, the built-in caching mechanism prevented subsequent queries to get the latest features.

Navigating Codebase

Both my Frontend and Backend code are in the same project, but under different root folders. Frontend JavaScript/TypeScript is under “src” and Backend PHP is under “php”. Database scripts are also in their own folder under “db”. I’m often getting lost scrolling the folder list going back-and forth between related backend and frontend files. I had debated on combining the two, so that PHP files would be intermingled with TypeScript files. However, this was going to lead to a nightmare pulling out the files during deployment since the names of my slices didn’t match the names of the API endpoints, and some slices worked with multiple endpoints.

The answer was markdown. Add a README.md file in the folder with relative links to other parts of the code base that are related to the folder you are in. You can link directly to files or folders. Command+Click opens it up for you. It’s better than symbolic links in that you can document where it takes you.

- [api](../../../php/api/)
  - [authentication](../../../php/api/authentication/README.md)
  - [authorization](../../../php/api/authorization/README.md)

Removing dead code

A couple days ago I decided it was time to drop the old Angular files that were in a backup folder, as I was no longer running to them for reference. I can always look at the Git history if I need something. Today I dropped the old API folder. It was based on fetch and had a built-in detection when the server was down, or had errors, to stop further requests and prevent overloading the server. RTK Query handles everything now.

Material Icon Picker

The navigation buttons have icons next to them. I needed a way to edit the icons. There are over 10,000 icons to choose from. I created a dialog that allows you to search icon names, as well as categorize them into groups. The filtering is fairly quick, but I ran into performance problems when rendering the grid. You could scroll through the icons, but there were just too many components rendered at once – and the vast majority were hidden. I didn’t want pagination as it’s better to scroll through the list. I needed some kind of virtual pagination with infinite scrolling.

History of the Web

Before I go on, let’s take a visit down memory lane regarding the different stages the World Wide Web (WWW) has gone through. Does anyone call it that anymore?

Comet

Comet was similar to AJAX in that it fetched data – but it kept the connection open so that the web server could continue to feed small bits of data over time. The best way to think about it, from a networking perspective, is to compare Transmission Control Protocol (TCP) vs User Datagram Protocol (UDP). One is oriented towards transferring a piece of data with quality, while the other is for streaming in that missed packets are fine and order of packets don’t matter. This is like downloading a file vs watching a live chat. Without comet, you could have to do polling to make multiple AJAX requests. Comet wasn’t used as often as it was fairly complex to implement. My JsonStreamer library is the server-side equivalent of a comet implementation, as it sends parts of a JSON object in small batches.

Comet has been superseded by web sockets.

Comet isn’t an acronym for anything. It came out at the same time as AJAX. I just remember growing up, my mother often had both AJAX and COMET powders for cleaning the bathroom.

Infinite Scrolling

So, with all of that, infinite scrolling was one of those buzz words that came out of Web 2.0. I was often fascinated by it. I understood the concepts from a backend perspective in that I just need to serve a page of records staring at an offset, limit the number of records returned, and provide the total number of matches. Howe was it done on the front-end though? What’s going on?

Today I sat down and got to the bottom of it. What your doing is displaying a small subset of data, but also adding padding before and after that subset, based on where the scroll position is currently located. To make things seamless, you had to load some data that was out of view so that you could smoothly scroll them into view while the next set of data was loaded.

The only way that you can pull this off is if you know the exact height of the window, the height of each row of data, and the total number of items that can match.

It worked out great and I had something setup quick and easy. Almost. You start running into issues when you start thinking about performance and a few other things.

Changing Tabs

As I changed between some tabs, I noticed that the scroll position wasn’t being set to the top. I gave the parent container the ability to pass in a ref to the scrollable container. However, the passing of the ref was optional, and I needed a ref within the component. Enter the function, useImperativeHandle.

import { forwardRef, useImperativeHandle, useRef } from 'react';
export const Foo = forwardRef<HTMLDivElement, {height?: number}>((
{ height = 400 }, ref) => {
  const localRef = useRef<HTMLDivElement(null);
  useImperativeHandle(ref, () => localRef.current!);
  return <div
    ref={localRef}
    style={{ height: `${height}px`, overflowY: 'auto'}}
    >
    <div style={margins}>
      {children}
    </div>
  </div>
}

The way that I have it setup, I am exposing the local element completely. I have the option to also limit what the parent component can call by providing a wrapper object instead of the element itself. At this point, each time the tab changed, the container component could call ref.current.scrollTo(0, 0) to scroll to the top.

Dependencies

Next I had a problem where two tabs had the same number of results. As I clicked between each tab, the onRender event wasn’t firing. The issue is that it only asks for content when the offset, limit, total, or onRender properties changed. To get around this, I added an option for the container to pass in a dependencies property. It could be anything. I set it up to a string that combined the two things that affected the results available:

${tab}:${lowerSearchQuery}

Deluge of Scroll Events

The scroll event happens constantly. If I scroll from the top of the page to the middle, no matter how fast I am (as a human), there are many events that fire as I drag the scroll bar to the middle. To circumvent this, you need to debounce the event so that if the user continues to scroll, you render after the latest debounced event fires. Once I did that, I ran into flickering problems.

The issue is that the margins are based on the first row of results, which is non-visible. I need the user to be able to see it scroll into view as I request additional rows. The top visible row is determined by the scroll position. It turns out a lot of math is involved in compensating for hidden rows, row height, scroll position, total rows, view window, etc.

Once I figured out what was going on, I stumbled upon Reacts hook to useDeferredValue. Debounce waits until you stop making a ton of repeated requests before it will call the function. Using a deferred value is similar, except that rather than waiting until the last request, it will run a rendering of the component with the latest changed value when it has time, compared to all other actions being dispatched. This means that I can load items while I’m still scrolling. I don’t have to let go of the scroll bar, or hold steady for the items to load. Given that it’s built into React, I don’t have to write my own debounce handling or import a third party library. Ideally, I’d like to avoid having too many third party libraries if functionality is simple enough to implement myself, and I don’t need all of the additional functionality that the other libraries provide.

const [scrollTop, setScrollTop] = useState(0);
// Calculations/Dependencies are based on deferredScrollTop
const deferredScrollTop = useDeferredValue(scrollTop);
const handleScroll: UIEventHandler<HTMLDivElement> = (event) => {
  setItemsOutOfView(true);
  setScrollTop(event.currentTarget.scrollTop);
}

Filter past last page

So the user scrolled down to the last page. Afterwards, they start typing into the search field, reducing the number of results. Suddenly we have a blank result set where we need to scroll up. There are two potential fixes for this. The easy fix is to scroll to the top any time the search text changes. The hard fix is to scroll the last items into view. Why is this important? You may be looking at items beginning with “Z”. If you start searching “Z”, you may not be interested in the prior results. This leads into smart filtering with focus retention, or a proximity search, which we will get into later. The fix was to calculate the scroll position of where the first row on the last page would have started.

Grouping Icons

10,607 icons. That’s just way too many icons to throw together. There needs to be some kind of grouping. They are based on Googles Material Design Icons, in which I browse through often looking for icons. Looking at the github: @google/material-design-icons, they do group the icons into several folders, but the names I have in Material UI don’t match up.

Upon further review, the Material UI has a small search filter for icons as well. There are only 2,128 icons at most. Most icon have a different style: filled, outlined, rounded, two-tone, and sharp. I added a radio group so that I could filter out the icons. I found some small differences, such as the Apple icon is only available with the Filled style.

It’s a bit hard to programmatically classify the icons for the appropriate style. In general, Icons end with the style, except for Filled icons.

• Edit (filled)
• EditOutlined
• EditRounded
• EditTwoTone
• EditSharp

Well, holy smokes. Why didn’t I think of this earlier. I decided to prompt a large language model to group the icons for me. It gave me a bulleted list for each category. Then I asked it to do the same, but format it as {Category: ['Icon1', 'Icon2']}. Wallah – I had my data.

Some of the icons don’t exist. I had typed my object so that it highlighted any names that were not in the Material UI Icons library. It provided many icons, but there were still 1,956 icons uncategorized. I pressed it further, which suggested that I use a library or programmatically parse out the names and come up with a categorization scheme. Finally, I got it to put out a larger list just by asking it to categorize as many as it could. It went so far that a button popped up that said “Continue Generating”. Later it seemed to hang. A refresh and simply stating “Continue” resulted in more groups.

The chat thought it was done. 1,241 icons remaining… Well, let’s just brute force this. I started spitting out a list of 100 icons each, and asked the model to categorize them. It was slowly working, but the AI was coming up with more groups. I was horizontally scrolling to see them all. I modified the tabs to wrap onto multiple lines. Normally MUI Tabs don’t support this, but you can pull it off by adding

sx={{
flexWrap:'wrap',
'.MuiTabs-flexContainer': {
flexWrap: 'wrap'
}}}

Ideally, we need to get the categories down to around seven groups that are one word. For now, let’s get the rest of our icons grouped, and then we can consolidate afterwards.

Sometimes the AI would give me odd names. Sometimes I found that the names were odd, such as ReportGmailerrorred. It feels like it should be named ReportGmailErrored. The model would still classify some icons as “General”, “Misc”, or “Miscellaneous”. I just went with it and let it chug away at the remaining icons in the “Other” category.

I got through everything for a rough draft. I now have seven categories. They are fairly broad. If I add any more categories, they are going to be sub-categories for a separate set of tabs to appear. However, I’m uncertain that I need it. I just needed to spread the icons out a bit so that it wasn’t one page of random icons.

Row Spacing

I noticed that I wasn’t seeing exactly four full rows in the scrollable viewport. Although the grid cells were exactly 100 pixels high, I realized that the containing grid was applying spacing. I went ahead and set the row spacing to zero. In addition, I decided to display more columns of icons. As a result, my earlier math isn’t mathing so well for scrolling, and has to be updated. Apparently I was making calculations against a side effect of row spacing.

This part of the project is almost done. I have over 2,000 items listed in my code file. That’s data – and thus should be move out of the source files. I’m debating if I should wire it up as a service call to grab a static JSON file or make the icons and their groupings managed in a database. Part of me is weighing the Keep It Simple Silly (KISS) principle. I could just create a separate file of the data and lazy load it when needed.

Anyhow, it’s almost 4:30 in the morning. I need to catch some sleep before making a trip tomorrow to visit and help out family. Witch means this problem is going to stew on my mind all weekend until I can go back home.

import {
  forwardRef,
  ReactNode,
  UIEventHandler,
  useDeferredValue,
  useEffect,
  useImperativeHandle,
  useMemo,
  useRef,
  useState
} from "react"

export const InfiniteScroll = forwardRef<HTMLDivElement, {
  height: number,
  totalItems: number,
  visibleRows: number,
  visibleColumns: number,
  dependencies?: any,
  onRender: (
    filter: { offset: number, limit: number }
  ) => ReactNode
}>(({
  height = 400,
  totalItems = 0,
  visibleRows = 1,
  visibleColumns = 1,
  onRender = () => null,
  dependencies
}, ref) => {

  const localRef = useRef<HTMLDivElement>(null);
  useImperativeHandle(ref, () => localRef.current!);

  const [scrollTop, setScrollTop] = useState(0);
  const deferredScrollTop = useDeferredValue(scrollTop);

  const [itemsOutOfView, setItemsOutOfView] = useState(false);

  const [margins, setMargins] = useState({
    marginTop: '0',
    height: 'unset'
  });

  const hiddenRowsBefore = 4;
  const hiddenRowsAfter = 4;
  const hiddenRows = hiddenRowsBefore + hiddenRowsAfter;

  const totalRows = Math.ceil(totalItems / visibleColumns);
  const pageHeight = 400;
  const rowHeight = pageHeight / visibleRows;
  const totalHeight = rowHeight * totalRows;
  const scrollableHeight = totalHeight - pageHeight;
  const scrollableRows = totalRows - hiddenRowsAfter;
  const visibleTopRow = Math.floor(
    (deferredScrollTop / scrollableHeight) * scrollableRows
  );
  const firstDataRow = visibleTopRow - hiddenRowsBefore;
  const offset = Math.max(0, firstDataRow * visibleColumns);
  const limit = (visibleRows + hiddenRows) * visibleColumns;
  const slidingHeight = (visibleRows + hiddenRows) * rowHeight;

  const children = useMemo(
    () => offset < totalItems ? onRender({ offset, limit }) : null,
    [offset, limit, totalItems, onRender, dependencies]
  );

  useEffect(() => {
    const marginTop = Math.max(0, firstDataRow) * rowHeight;
    setMargins({
      marginTop: `${marginTop}px`,
      height: `${(totalHeight)}px`
    });
    setItemsOutOfView(false);
  }, [
    firstDataRow,
    rowHeight,
    totalHeight
  ]);

  const scrollLastPageIntoView = () => {
    const y = (totalRows - visibleRows) * rowHeight;
    if (deferredScrollTop > y) {
      localRef.current?.scrollTo(0, y);
    }
  }
  useEffect(() => {
    if (offset >= totalItems - (visibleRows * visibleColumns)) {
      scrollLastPageIntoView();
    }
  }, [offset, totalItems, visibleRows, visibleColumns])


  const handleScroll: UIEventHandler<HTMLDivElement> = (event) => {
    setItemsOutOfView(true);
    setScrollTop(event.currentTarget.scrollTop);
  }
  return <div
    ref={localRef}
    style={{
      height: `${height}px`,
      overflowY: 'auto',
      transition: 'background-color 1s ease',
      backgroundColor: itemsOutOfView ? 'rgba(0,0,0,.5)' : ''
    }}
    onScroll={handleScroll}
  >
    <div style={{
      ...margins,
      overflow: 'hidden',
      height: `${slidingHeight}px`
    }}>
      {children}
    </div>
  </div>
})