AEM: Dialog Editor For Classic UI

Dialog Editor
The dialog editor provides a graphical interface for easily creating and editing dialog boxes and scaffolds. To see how it works, double click on the component's dialog.

In Dilaog Editor you can build your dialog with some cq provided widgets like textfield, pathfield, etc...

You may face the following issue

1)  Unable to see dialog after double click on it when any custom widget configured 
For this, we have to do overlay html.jsp.

Copy html.jsp form /libs/foundation/components/primary/cq/Dialog/html.jsp to /apps/foundation/components/primary/cq/Dialog/html.jsp

In html.jsp at line number 32, you can see  client libs include to that add your custom widgets client libs 

<cq:includeClientLib categories="cq.widgets,cq.tagging,cq.scaffolding,custom.widgets"/>

2)  Unable to find custom widgets in the palette to build dialog
For this, you have to overlay "DialogEditorConstants.js". Copy /libs/cq/ui/widgets/source/widgets/DialogEditorConstants.js to  /apps/cq/ui/widgets/source/widgets/DialogEditorConstants.js. After the copy, you have to configure your custom widgets / cq widgets.

To Configure follow these steps 
Add your custom/cq provided xtype to CQ.DialogEditor.ALLOWEDCHILDS array 

Eg: Here  I have added dialogfieldset

CQ.DialogEditor.ALLOWEDCHILDS = [ 
            "textfield", 
            "textarea", 
            "numberfield", 
            "selection", 
            "combo", 
            "hidden",
            "richtext",
     "multifield",
     "checkbox",
     "pathfield",
      "dialogfieldset"
  ];

Add your custom/cq provided xtype configs to CQ.DialogEditor.MAPPINGS object

add the below object

dialogfieldset: {
        category:CQ.DialogEditor.CONTAINER,
        text:CQ.I18n.getMessage("Dialogfieldset"),
        insertCfg:{
            "jcr:primaryType":"cq:Widget",
            xtype:"dialogfieldset",
            items:{ "jcr:primaryType":"cq:WidgetCollection" }
        },
  allowedChilds: CQ.DialogEditor.ALLOWEDCHILDS,
        propList:{
            title:"Dialogfieldset",
            width:"",
            height:"",
   collapsed : false,
   collapsible: true,
   id : ""
        }
    }

you can add some more properties also if u want. you can see the dialog 'dialogfieldset' under Containers Palette

References
Adobe Document