Merge PR from JeLuF/gallery

3rd-PARTY-LICENSES

@@ -712,3 +712,31 @@ FileSaver.js is licensed under the MIT license:
     SOFTWARE.
     
       [1]: http://eligrey.com
+
+croppr.js
+=========
+https://github.com/jamesssooi/Croppr.js
+
+croppr.js is licensed under the MIT license:
+
+    MIT License
+
+    Copyright (c) 2017 James Ooi
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE.

CHANGES.md

@@ -22,6 +22,8 @@
 Our focus continues to remain on an easy installation experience, and an easy user-interface. While still remaining pretty powerful, in terms of features and speed.
 
 ### Detailed changelog
+* 3.0.0 - 3 Aug 2023 - Enabled diffusers for everyone by default. The old v2 engine can be used by disabling the "Use v3 engine" option in the Settings tab.
+* 2.5.48 - 1 Aug 2023 - (beta-only) Full support for ControlNets. You can select a control image to guide the AI. You can pick a filter to pre-process the image, and one of the known (or custom) controlnet models. Supports `OpenPose`, `Canny`, `Straight Lines`, `Depth`, `Line Art`, `Scribble`, `Soft Edge`, `Shuffle` and `Segment`.
 * 2.5.47 - 30 Jul 2023 - An option to use `Strict Mask Border` while inpainting, to avoid touching areas outside the mask. But this might show a slight outline of the mask, which you will have to touch up separately.
 * 2.5.47 - 29 Jul 2023 - (beta-only) Fix long prompts with SDXL.
 * 2.5.47 - 29 Jul 2023 - (beta-only) Fix red dots in some SDXL images.

How to install and run.txt

@@ -5,10 +5,10 @@ If you haven't downloaded Stable Diffusion UI yet, please download from https://
 After downloading, to install please follow these instructions:
 
 For Windows:
-- Please double-click the "Start Stable Diffusion UI.cmd" file inside the "stable-diffusion-ui" folder.
+- Please double-click the "Easy-Diffusion-Windows.exe" file and follow the instructions.
 
 For Linux:
-- Please open a terminal, and go to the "stable-diffusion-ui" directory. Then run ./start.sh
+- Please open a terminal, unzip the Easy-Diffusion-Linux.zip file and go to the "easy-diffusion" directory. Then run ./start.sh
 
 That file will automatically install everything. After that it will start the Stable Diffusion interface in a web browser.
 

README.md

@@ -11,9 +11,9 @@ Does not require technical knowledge, does not require pre-installed software. 1
 Click the download button for your operating system:
 
 <p float="left">
-  <a href="https://github.com/easydiffusion/easydiffusion/releases/download/v2.5.24/Easy-Diffusion-Windows.exe"><img src="https://github.com/easydiffusion/easydiffusion/raw/main/media/download-win.png" width="200" /></a>
+  <a href="https://github.com/easydiffusion/easydiffusion/releases/download/v2.5.41a/Easy-Diffusion-Windows.exe"><img src="https://github.com/easydiffusion/easydiffusion/raw/main/media/download-win.png" width="200" /></a>
-  <a href="https://github.com/easydiffusion/easydiffusion/releases/download/v2.5.24/Easy-Diffusion-Linux.zip"><img src="https://github.com/easydiffusion/easydiffusion/raw/main/media/download-linux.png" width="200" /></a>
+  <a href="https://github.com/easydiffusion/easydiffusion/releases/download/v2.5.41a/Easy-Diffusion-Linux.zip"><img src="https://github.com/easydiffusion/easydiffusion/raw/main/media/download-linux.png" width="200" /></a>
-  <a href="https://github.com/easydiffusion/easydiffusion/releases/download/v2.5.24/Easy-Diffusion-Mac.zip"><img src="https://github.com/easydiffusion/easydiffusion/raw/main/media/download-mac.png" width="200" /></a>
+  <a href="https://github.com/easydiffusion/easydiffusion/releases/download/v2.5.41a/Easy-Diffusion-Mac.zip"><img src="https://github.com/easydiffusion/easydiffusion/raw/main/media/download-mac.png" width="200" /></a>
 </p>
 
 **Hardware requirements:**
@@ -23,6 +23,7 @@ Click the download button for your operating system:
 - Minimum 8 GB of system RAM.
 - Atleast 25 GB of space on the hard disk.
 
+
 The installer will take care of whatever is needed. If you face any problems, you can join the friendly [Discord community](https://discord.com/invite/u9yhsFmEkB) and ask for assistance.
 
 ## On Windows:
@@ -132,6 +133,15 @@ We could really use help on these aspects (click to view tasks that need your he
 
 If you have any code contributions in mind, please feel free to say Hi to us on the [discord server](https://discord.com/invite/u9yhsFmEkB). We use the Discord server for development-related discussions, and for helping users.
 
+# Credits
+* Stable Diffusion: https://github.com/Stability-AI/stablediffusion
+* CodeFormer: https://github.com/sczhou/CodeFormer (license: https://github.com/sczhou/CodeFormer/blob/master/LICENSE)
+* GFPGAN: https://github.com/TencentARC/GFPGAN
+* RealESRGAN: https://github.com/xinntao/Real-ESRGAN
+* k-diffusion: https://github.com/crowsonkb/k-diffusion
+* Code contributors and artists on the cmdr2 UI: https://github.com/cmdr2/stable-diffusion-ui and Discord (https://discord.com/invite/u9yhsFmEkB)
+* Lots of contributors on the internet
+
 # Disclaimer
 The authors of this project are not responsible for any content generated using this interface.
 

scripts/check_modules.py

@@ -18,7 +18,7 @@ os_name = platform.system()
 modules_to_check = {
     "torch": ("1.11.0", "1.13.1", "2.0.0"),
     "torchvision": ("0.12.0", "0.14.1", "0.15.1"),
-    "sdkit": "1.0.151",
+    "sdkit": "1.0.167",
     "stable-diffusion-sdkit": "2.1.4",
     "rich": "12.6.0",
     "uvicorn": "0.19.0",

ui/easydiffusion/app.py

@@ -61,6 +61,7 @@ APP_CONFIG_DEFAULTS = {
     "ui": {
         "open_browser_on_start": True,
     },
+    "test_diffusers": True,
 }
 
 IMAGE_EXTENSIONS = [
@@ -116,7 +117,7 @@ def getConfig(default_val=APP_CONFIG_DEFAULTS):
 
     def set_config_on_startup(config: dict):
         if getConfig.__test_diffusers_on_startup is None:
-            getConfig.__test_diffusers_on_startup = config.get("test_diffusers", False)
+            getConfig.__test_diffusers_on_startup = config.get("test_diffusers", True)
         config["config_on_startup"] = {"test_diffusers": getConfig.__test_diffusers_on_startup}
 
     if os.path.isfile(config_yaml_path):

ui/easydiffusion/bucket_manager.py

@@ -71,8 +71,7 @@ def init():
         bucket = crud.get_bucket_by_path(db, path)
 
         if bucket == None:
-            bucket_id = crud.create_bucket(db=db, bucket=schemas.BucketCreate(path=path))
+            bucket = crud.create_bucket(db=db, bucket=schemas.BucketCreate(path=path))
-        else:
         bucket_id = bucket.id
 
         bucketfile = schemas.BucketFileCreate(filename=filename, data=file)
@@ -92,25 +91,19 @@ def init():
 
     @server_api.get("/image/{image_path:path}")
     def get_image(image_path: str, db: Session = Depends(get_db)):
-        from easydiffusion.easydb.mappings import Image
+        from easydiffusion.easydb.mappings import GalleryImage
         image_path = str(abspath(image_path))
-        amount = len(db.query(Image).filter(Image.path == image_path).all())
+        try:
-        if amount > 0:
+            image = db.query(GalleryImage).filter(GalleryImage.path == image_path).first()
-            image = db.query(Image).filter(Image.path == image_path).first()
             return FileResponse(image.path)
-        else:
+        except Exception as e:
             raise HTTPException(status_code=404, detail="Image not found")
     
     @server_api.get("/all_images")
     def get_all_images(db: Session = Depends(get_db)):
-        from easydiffusion.easydb.mappings import Image
+        from easydiffusion.easydb.mappings import GalleryImage
-        images = db.query(Image).all()
+        images = db.query(GalleryImage).all()
-        sum_string = "<div id='imagecontainer'>"
+        return images
-        for img in images:
-            options = f"Path: {img.path}\nPrompt: {img.prompt}\nNegative Prompt: {img.negative_prompt}\nSeed: {img.seed}\nModel: {img.use_stable_diffusion_model}\nSize: {img.height}x{img.width}\nSampler: {img.sampler_name}\nSteps: {img.num_inference_steps}\nGuidance Scale: {img.guidance_scale}\nLoRA: {img.lora}\nUpscaling: {img.use_upscale}\nFace Correction: {img.use_face_correction}\n"
-            sum_string += f"<img src='/image/{img.path}' title='{options}'>"
-        sum_string += "</div>"
-        return Response(content=sum_string, media_type="text/html")
 
 
 def get_filename_from_url(url):

ui/easydiffusion/easydb/crud.py

@@ -19,7 +19,6 @@ def create_bucketfile(db: Session, bucketfile: schemas.BucketFileCreate, bucket_
     db_bucketfile = models.BucketFile(**bucketfile.dict(), bucket_id=bucket_id)
     db.merge(db_bucketfile)
     db.commit()
-    from pprint import pprint
     db_bucketfile = db.query(models.BucketFile).filter(models.BucketFile.bucket_id==bucket_id, models.BucketFile.filename==bucketfile.filename).first()
     return db_bucketfile
 

ui/easydiffusion/easydb/database.py

@@ -7,7 +7,6 @@ from sqlalchemy.orm import sessionmaker
 
 os.makedirs(app.BUCKET_DIR, exist_ok=True)
 SQLALCHEMY_DATABASE_URL = "sqlite:///"+os.path.join(app.BUCKET_DIR, "bucket.db")
-print("## SQLALCHEMY_DATABASE_URL = ", SQLALCHEMY_DATABASE_URL)
 
 engine = create_engine(SQLALCHEMY_DATABASE_URL, connect_args={"check_same_thread": False})
 SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)

ui/easydiffusion/easydb/mappings.py

@@ -1,9 +1,10 @@
-from sqlalchemy import Column, Integer, String, Float, Boolean
+from sqlalchemy import Column, Integer, String, Float, Boolean, DateTime
 from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.sql import func
 
 Base = declarative_base()
 
-class Image(Base):
+class GalleryImage(Base):
     __tablename__ = 'images'
 
     path = Column(String, primary_key=True)
@@ -23,10 +24,11 @@ class Image(Base):
     use_upscale = Column(String)
     prompt = Column(String)
     negative_prompt = Column(String)
+    time_created = Column(DateTime(timezone=True), server_default=func.now())
 
     def __repr__(self):
-        return "<Image(path='%s', seed='%s', use_stable_diffusion_model='%s', clip_skip='%s', use_vae_model='%s', sampler_name='%s', width='%s', height='%s', num_inference_steps='%s', guidance_scale='%s', lora='%s', use_hypernetwork_model='%s', tiling='%s', use_face_correction='%s', use_upscale='%s', prompt='%s', negative_prompt='%s')>" % (
+        return "<GalleryImage(path='%s', seed='%s', use_stable_diffusion_model='%s', clip_skip='%s', use_vae_model='%s', sampler_name='%s', width='%s', height='%s', num_inference_steps='%s', guidance_scale='%s', lora='%s', use_hypernetwork_model='%s', tiling='%s', use_face_correction='%s', use_upscale='%s', prompt='%s', negative_prompt='%s')>" % (
             self.path, self.seed, self.use_stable_diffusion_model, self.clip_skip, self.use_vae_model, self.sampler_name, self.width, self.height, self.num_inference_steps, self.guidance_scale, self.lora, self.use_hypernetwork_model, self.tiling, self.use_face_correction, self.use_upscale, self.prompt, self.negative_prompt)
 
 from easydiffusion.easydb.database import engine
-Image.metadata.create_all(engine)
+GalleryImage.metadata.create_all(engine)

ui/easydiffusion/model_manager.py

@@ -9,6 +9,7 @@ from easydiffusion.types import ModelsData
 from easydiffusion.utils import log
 from sdkit import Context
 from sdkit.models import load_model, scan_model, unload_model, download_model, get_model_info_from_db
+from sdkit.models.model_loader.controlnet_filters import filters as cn_filters
 from sdkit.utils import hash_file_quick
 
 KNOWN_MODEL_TYPES = [
@@ -19,6 +20,8 @@ KNOWN_MODEL_TYPES = [
     "realesrgan",
     "lora",
     "codeformer",
+    "embeddings",
+    "controlnet",
 ]
 MODEL_EXTENSIONS = {
     "stable-diffusion": [".ckpt", ".safetensors"],
@@ -29,6 +32,7 @@ MODEL_EXTENSIONS = {
     "lora": [".ckpt", ".safetensors"],
     "codeformer": [".pth"],
     "embeddings": [".pt", ".bin", ".safetensors"],
+    "controlnet": [".pth", ".safetensors"],
 }
 DEFAULT_MODELS = {
     "stable-diffusion": [
@@ -144,7 +148,7 @@ def reload_models_if_necessary(context: Context, models_data: ModelsData, models
     models_to_reload = {
         model_type: path
         for model_type, path in models_data.model_paths.items()
-        if context.model_paths.get(model_type) != path
+        if context.model_paths.get(model_type) != path or (path is not None and context.models.get(model_type) is None)
     }
 
     if models_data.model_paths.get("codeformer"):
@@ -177,10 +181,17 @@ def reload_models_if_necessary(context: Context, models_data: ModelsData, models
 def resolve_model_paths(models_data: ModelsData):
     model_paths = models_data.model_paths
     for model_type in model_paths:
-        if model_type in ("latent_upscaler", "nsfw_checker"):  # doesn't use model paths
+        skip_models = cn_filters + ["latent_upscaler", "nsfw_checker"]
+        if model_type in skip_models:  # doesn't use model paths
             continue
         if model_type == "codeformer":
             download_if_necessary("codeformer", "codeformer.pth", "codeformer-0.1.0")
+        elif model_type == "controlnet":
+            model_id = model_paths[model_type]
+            model_info = get_model_info_from_db(model_type=model_type, model_id=model_id)
+            if model_info:
+                filename = model_info.get("url", "").split("/")[-1]
+                download_if_necessary("controlnet", filename, model_id, skip_if_others_exist=False)
 
         model_paths[model_type] = resolve_model_to_use(model_paths[model_type], model_type=model_type)
 
@@ -204,17 +215,17 @@ def download_default_models_if_necessary():
         print(model_type, "model(s) found.")
 
 
-def download_if_necessary(model_type: str, file_name: str, model_id: str):
+def download_if_necessary(model_type: str, file_name: str, model_id: str, skip_if_others_exist=True):
     model_path = os.path.join(app.MODELS_DIR, model_type, file_name)
     expected_hash = get_model_info_from_db(model_type=model_type, model_id=model_id)["quick_hash"]
 
-    other_models_exist = any_model_exists(model_type)
+    other_models_exist = any_model_exists(model_type) and skip_if_others_exist
     known_model_exists = os.path.exists(model_path)
     known_model_is_corrupt = known_model_exists and hash_file_quick(model_path) != expected_hash
 
     if known_model_is_corrupt or (not other_models_exist and not known_model_exists):
         print("> download", model_type, model_id)
-        download_model(model_type, model_id, download_base_dir=app.MODELS_DIR)
+        download_model(model_type, model_id, download_base_dir=app.MODELS_DIR, download_config_if_available=False)
 
 
 def migrate_legacy_model_location():
@@ -285,12 +296,26 @@ def is_malicious_model(file_path):
 def getModels(scan_for_malicious: bool = True):
     models = {
         "options": {
-            "stable-diffusion": ["sd-v1-4"],
+            "stable-diffusion": [{"sd-v1-4": "SD 1.4"}],
             "vae": [],
             "hypernetwork": [],
             "lora": [],
-            "codeformer": ["codeformer"],
+            "codeformer": [{"codeformer": "CodeFormer"}],
             "embeddings": [],
+            "controlnet": [
+                {"control_v11p_sd15_canny": "Canny (*)"},
+                {"control_v11p_sd15_openpose": "OpenPose (*)"},
+                {"control_v11p_sd15_normalbae": "Normal BAE (*)"},
+                {"control_v11f1p_sd15_depth": "Depth (*)"},
+                {"control_v11p_sd15_scribble": "Scribble"},
+                {"control_v11p_sd15_softedge": "Soft Edge"},
+                {"control_v11p_sd15_inpaint": "Inpaint"},
+                {"control_v11p_sd15_lineart": "Line Art"},
+                {"control_v11p_sd15s2_lineart_anime": "Line Art Anime"},
+                {"control_v11p_sd15_mlsd": "Straight Lines"},
+                {"control_v11p_sd15_seg": "Segment"},
+                {"control_v11e_sd15_shuffle": "Shuffle"},
+            ],
         },
     }
 
@@ -299,9 +324,9 @@ def getModels(scan_for_malicious: bool = True):
     class MaliciousModelException(Exception):
         "Raised when picklescan reports a problem with a model"
 
-    def scan_directory(directory, suffixes, directoriesFirst: bool = True):
+    def scan_directory(directory, suffixes, directoriesFirst: bool = True, default_entries=[]):
+        tree = list(default_entries)
         nonlocal models_scanned
-        tree = []
         for entry in sorted(
             os.scandir(directory),
             key=lambda entry: (entry.is_file() == directoriesFirst, entry.name.lower()),
@@ -320,7 +345,14 @@ def getModels(scan_for_malicious: bool = True):
                         raise MaliciousModelException(entry.path)
                 if scan_for_malicious:
                     known_models[entry.path] = mtime
-                tree.append(entry.name[: -len(matching_suffix)])
+                model_id = entry.name[: -len(matching_suffix)]
+                model_exists = False
+                for m in tree:  # allows default "named" models, like CodeFormer and known ControlNet models
+                    if (isinstance(m, str) and model_id == m) or (isinstance(m, dict) and model_id in m):
+                        model_exists = True
+                        break
+                if not model_exists:
+                    tree.append(model_id)
             elif entry.is_dir():
                 scan = scan_directory(entry.path, suffixes, directoriesFirst=False)
 
@@ -337,7 +369,8 @@ def getModels(scan_for_malicious: bool = True):
             os.makedirs(models_dir)
 
         try:
-            models["options"][model_type] = scan_directory(models_dir, model_extensions)
+            default_tree = models["options"].get(model_type, [])
+            models["options"][model_type] = scan_directory(models_dir, model_extensions, default_entries=default_tree)
         except MaliciousModelException as e:
             models["scan-error"] = str(e)
 
@@ -350,6 +383,7 @@ def getModels(scan_for_malicious: bool = True):
     listModels(model_type="gfpgan")
     listModels(model_type="lora")
     listModels(model_type="embeddings")
+    listModels(model_type="controlnet")
 
     if scan_for_malicious and models_scanned > 0:
         log.info(f"[green]Scanned {models_scanned} models. Nothing infected[/]")

ui/easydiffusion/package_manager.py

@@ -12,9 +12,9 @@ from easydiffusion import app
 manifest = {
     "tensorrt": {
         "install": [
-            "nvidia-cudnn --extra-index-url=https://pypi.ngc.nvidia.com --trusted-host pypi.ngc.nvidia.com",
+            "nvidia-cudnn --pre --extra-index-url=https://pypi.ngc.nvidia.com --trusted-host pypi.ngc.nvidia.com",
-            "tensorrt-libs --extra-index-url=https://pypi.ngc.nvidia.com --trusted-host pypi.ngc.nvidia.com",
+            "tensorrt-libs --pre --extra-index-url=https://pypi.ngc.nvidia.com --trusted-host pypi.ngc.nvidia.com",
-            "tensorrt --extra-index-url=https://pypi.ngc.nvidia.com --trusted-host pypi.ngc.nvidia.com",
+            "tensorrt --pre --extra-index-url=https://pypi.ngc.nvidia.com --trusted-host pypi.ngc.nvidia.com",
         ],
         "uninstall": ["tensorrt"],
         # TODO also uninstall tensorrt-libs and nvidia-cudnn, but do it upon restarting (avoid 'file in use' error)
@@ -25,7 +25,7 @@ installing = []
 # remove this once TRT releases on pypi
 if platform.system() == "Windows":
     trt_dir = os.path.join(app.ROOT_DIR, "tensorrt")
-    if os.path.exists(trt_dir):
+    if os.path.exists(trt_dir) and os.path.isdir(trt_dir) and len(os.listdir(trt_dir)) > 0:
         files = os.listdir(trt_dir)
 
         packages = manifest["tensorrt"]["install"]
@@ -61,6 +61,10 @@ def install(module_name):
         raise RuntimeError(f"Can't install unknown package: {module_name}!")
 
     commands = manifest[module_name]["install"]
+    if module_name == "tensorrt":
+        commands += [
+            "protobuf==3.20.3 polygraphy==0.47.1 onnx==1.14.0 --extra-index-url=https://pypi.ngc.nvidia.com --trusted-host pypi.ngc.nvidia.com"
+        ]
     commands = [f"python -m pip install --upgrade {cmd}" for cmd in commands]
 
     installing.append(module_name)

ui/easydiffusion/runtime.py

@@ -31,7 +31,7 @@ def init(device):
 
     app_config = app.getConfig()
     context.test_diffusers = (
-        app_config.get("test_diffusers", False) and app_config.get("update_branch", "main") != "main"
+        app_config.get("test_diffusers", True) and app_config.get("update_branch", "main") != "main"
     )
 
     log.info("Device usage during initialization:")

ui/easydiffusion/server.py

@@ -63,7 +63,7 @@ class SetAppConfigRequest(BaseModel, extra=Extra.allow):
     ui_open_browser_on_start: bool = None
     listen_to_network: bool = None
     listen_port: int = None
-    test_diffusers: bool = False
+    test_diffusers: bool = True
 
 
 def init():

ui/easydiffusion/tasks/render_images.py

@@ -15,6 +15,7 @@ from sdkit.utils import (
     img_to_base64_str,
     img_to_buffer,
     latent_samples_to_images,
+    log,
 )
 
 from .task import Task
@@ -63,7 +64,7 @@ class RenderTask(Task):
         if (
             runtime.set_vram_optimizations(context)
             or self.has_param_changed(context, "clip_skip")
-            or self.has_param_changed(context, "convert_to_tensorrt")
+            or self.trt_needs_reload(context)
         ):
             models_to_force_reload.append("stable-diffusion")
 
@@ -92,6 +93,29 @@ class RenderTask(Task):
         new_val = self.models_data.model_params.get("stable-diffusion", {}).get(param_name, False)
         return model["params"].get(param_name) != new_val
 
+    def trt_needs_reload(self, context):
+        if not context.test_diffusers:
+            return False
+        if "stable-diffusion" not in context.models or "params" not in context.models["stable-diffusion"]:
+            return True
+
+        model = context.models["stable-diffusion"]
+
+        # curr_convert_to_trt = model["params"].get("convert_to_tensorrt")
+        new_convert_to_trt = self.models_data.model_params.get("stable-diffusion", {}).get("convert_to_tensorrt", False)
+
+        pipe = model["default"]
+        is_trt_loaded = hasattr(pipe.unet, "_allocate_trt_buffers") or hasattr(
+            pipe.unet, "_allocate_trt_buffers_backup"
+        )
+        if new_convert_to_trt and not is_trt_loaded:
+            return True
+
+        curr_build_config = model["params"].get("trt_build_config")
+        new_build_config = self.models_data.model_params.get("stable-diffusion", {}).get("trt_build_config", {})
+
+        return new_convert_to_trt and curr_build_config != new_build_config
+
 
 def make_images(
     context,
@@ -148,6 +172,7 @@ def make_images_internal(
         context,
         req,
         task_data,
+        models_data,
         data_queue,
         task_temp_images,
         step_callback,
@@ -174,6 +199,7 @@ def generate_images_internal(
     context,
     req: GenerateImageRequest,
     task_data: TaskData,
+    models_data: ModelsData,
     data_queue: queue.Queue,
     task_temp_images: list,
     step_callback,
@@ -197,6 +223,30 @@ def generate_images_internal(
         if req.init_image is not None and not context.test_diffusers:
             req.sampler_name = "ddim"
 
+        req.width, req.height = map(lambda x: x - x % 8, (req.width, req.height))  # clamp to 8
+
+        if req.control_image and task_data.control_filter_to_apply:
+            req.control_image = filter_images(context, req.control_image, task_data.control_filter_to_apply)[0]
+
+        if context.test_diffusers:
+            pipe = context.models["stable-diffusion"]["default"]
+            if hasattr(pipe.unet, "_allocate_trt_buffers_backup"):
+                setattr(pipe.unet, "_allocate_trt_buffers", pipe.unet._allocate_trt_buffers_backup)
+                delattr(pipe.unet, "_allocate_trt_buffers_backup")
+
+            if hasattr(pipe.unet, "_allocate_trt_buffers"):
+                convert_to_trt = models_data.model_params["stable-diffusion"].get("convert_to_tensorrt", False)
+                if convert_to_trt:
+                    pipe.unet.forward = pipe.unet._trt_forward
+                    # pipe.vae.decoder.forward = pipe.vae.decoder._trt_forward
+                    log.info(f"Setting unet.forward to TensorRT")
+                else:
+                    log.info(f"Not using TensorRT for unet.forward")
+                    pipe.unet.forward = pipe.unet._non_trt_forward
+                    # pipe.vae.decoder.forward = pipe.vae.decoder._non_trt_forward
+                    setattr(pipe.unet, "_allocate_trt_buffers_backup", pipe.unet._allocate_trt_buffers)
+                    delattr(pipe.unet, "_allocate_trt_buffers")
+
         images = generate_images(context, callback=callback, **req.dict())
         user_stopped = False
     except UserInitiatedStop:

ui/easydiffusion/types.py

@@ -75,6 +75,7 @@ class TaskData(BaseModel):
     use_controlnet_model: Union[str, List[str]] = None
     filters: List[str] = []
     filter_params: Dict[str, Dict[str, Any]] = {}
+    control_filter_to_apply: Union[str, List[str]] = None
 
     show_only_filtered_image: bool = False
     block_nsfw: bool = False
@@ -135,6 +136,7 @@ class GenerateImageResponse:
     def json(self):
         del self.render_request.init_image
         del self.render_request.init_image_mask
+        del self.render_request.control_image
 
         task_data = self.task_data.dict()
         task_data.update(self.output_format.dict())
@@ -212,6 +214,9 @@ def convert_legacy_render_req_to_new(old_req: dict):
     model_paths["latent_upscaler"] = (
         model_paths["latent_upscaler"] if "latent_upscaler" in model_paths["latent_upscaler"].lower() else None
     )
+    if "control_filter_to_apply" in old_req:
+        filter_model = old_req["control_filter_to_apply"]
+        model_paths[filter_model] = filter_model
 
     if old_req.get("block_nsfw"):
         model_paths["nsfw_checker"] = "nsfw_checker"
@@ -221,6 +226,9 @@ def convert_legacy_render_req_to_new(old_req: dict):
         model_params["stable-diffusion"] = {
             "clip_skip": bool(old_req.get("clip_skip", False)),
             "convert_to_tensorrt": bool(old_req.get("convert_to_tensorrt", False)),
+            "trt_build_config": old_req.get(
+                "trt_build_config", {"batch_size_range": (1, 1), "dimensions_range": [(768, 1024)]}
+            ),
         }
 
     # move the filter params

ui/easydiffusion/utils/save_utils.py

@@ -21,6 +21,8 @@ TASK_TEXT_MAPPING = {
     "seed": "Seed",
     "use_stable_diffusion_model": "Stable Diffusion model",
     "clip_skip": "Clip Skip",
+    "use_controlnet_model": "ControlNet model",
+    "control_filter_to_apply": "ControlNet Filter",
     "use_vae_model": "VAE model",
     "sampler_name": "Sampler",
     "width": "Width",
@@ -155,11 +157,11 @@ def save_images_to_disk(
                 else:
                     return metadata_entries[i]["use_lora_model"] + ":" + str(metadata_entries[i]["lora_alpha"])
 
-            from easydiffusion.easydb.mappings import Image
+            from easydiffusion.easydb.mappings import GalleryImage
             from easydiffusion.easydb.database import SessionLocal
 
             session = SessionLocal()
-            session.add(Image(
+            session.add(GalleryImage(
                 path = path_i,
                 seed = metadata_entries[i]["seed"],
                 use_stable_diffusion_model = metadata_entries[i]["use_stable_diffusion_model"],
@@ -258,7 +260,7 @@ def get_printable_request(req: GenerateImageRequest, task_data: TaskData, output
     task_data_metadata.update(output_format.dict())
 
     app_config = app.getConfig()
-    using_diffusers = app_config.get("test_diffusers", False)
+    using_diffusers = app_config.get("test_diffusers", True)
 
     # Save the metadata in the order defined in TASK_TEXT_MAPPING
     metadata = {}
@@ -301,10 +303,12 @@ def get_printable_request(req: GenerateImageRequest, task_data: TaskData, output
         del metadata["lora_alpha"]
     if task_data.use_upscale != "latent_upscaler" and "latent_upscaler_steps" in metadata:
         del metadata["latent_upscaler_steps"]
+    if task_data.use_controlnet_model is None and "control_filter_to_apply" in metadata:
+        del metadata["control_filter_to_apply"]
 
     if not using_diffusers:
         for key in (
-            x for x in ["use_lora_model", "lora_alpha", "clip_skip", "tiling", "latent_upscaler_steps"] if x in metadata
+            x for x in ["use_lora_model", "lora_alpha", "clip_skip", "tiling", "latent_upscaler_steps", "use_controlnet_model", "control_filter_to_apply"] if x in metadata
         ):
             del metadata[key]
 

ui/index.html

@@ -18,12 +18,14 @@
     <link rel="stylesheet" href="/media/css/image-modal.css">
     <link rel="stylesheet" href="/media/css/plugins.css">
     <link rel="stylesheet" href="/media/css/animations.css">
+    <link rel="stylesheet" href="/media/css/croppr.css" rel="stylesheet"/>
     <link rel="manifest" href="/media/manifest.webmanifest">
     <script src="/media/js/jquery-3.6.1.min.js"></script>
     <script src="/media/js/jquery-confirm.min.js"></script>
     <script src="/media/js/jszip.min.js"></script>
     <script src="/media/js/FileSaver.min.js"></script>
     <script src="/media/js/marked.min.js"></script>
+    <script src="/media/js/croppr.js"></script>
 </head>
 <body>
 <div id="container">
@@ -32,7 +34,7 @@
             <h1>
                 <img id="logo_img" src="/media/images/icon-512x512.png" >
                 Easy Diffusion
-                <small><span id="version">v2.5.47</span> <span id="updateBranchLabel"></span></small>
+                <small><span id="version">v3.0.0</span> <span id="updateBranchLabel"></span></small>
             </h1>
         </div>
         <div id="server-status">
@@ -86,8 +88,8 @@
                     <label for="init_image">Initial Image (img2img) <small>(optional)</small> </label>
 
                     <div id="init_image_preview_container" class="image_preview_container">
-                        <div id="init_image_wrapper">
+                        <div id="init_image_wrapper" class="preview_image_wrapper">
-                            <img id="init_image_preview" src="" crossorigin="anonymous" />
+                            <img id="init_image_preview" class="image_preview" src="" crossorigin="anonymous" />
                             <span id="init_image_size_box" class="img_bottom_label"></span>
                             <button class="init_image_clear image_clear_btn"><i class="fa-solid fa-xmark"></i></button>
                         </div>
@@ -144,18 +146,17 @@
                     <div><table>
                     <tr><b class="settings-subheader">Image Settings</b></tr>
                     <tr class="pl-5"><td><label for="seed">Seed:</label></td><td><input id="seed" name="seed" size="10" value="0" onkeypress="preventNonNumericalInput(event)"> <input id="random_seed" name="random_seed" type="checkbox" checked><label for="random_seed">Random</label></td></tr>
-                    <tr class="pl-5"><td><label for="num_outputs_total">Number of Images:</label></td><td><input id="num_outputs_total" name="num_outputs_total" value="1" size="1" onkeypress="preventNonNumericalInput(event)"> <label><small>(total)</small></label> <input id="num_outputs_parallel" name="num_outputs_parallel" value="1" size="1" onkeypress="preventNonNumericalInput(event)"> <label for="num_outputs_parallel"><small>(in parallel)</small></label></td></tr>
+                    <tr class="pl-5"><td><label for="num_outputs_total">Number of Images:</label></td><td><input id="num_outputs_total" name="num_outputs_total" value="1" size="1" onkeypress="preventNonNumericalInput(event)"> <label><small>(total)</small></label> <input id="num_outputs_parallel" name="num_outputs_parallel" value="1" size="1" onkeypress="preventNonNumericalInput(event)"> <label id="num_outputs_parallel_label" for="num_outputs_parallel"><small>(in parallel)</small></label></td></tr>
                     <tr class="pl-5"><td><label for="stable_diffusion_model">Model:</label></td><td class="model-input">
                         <input id="stable_diffusion_model" type="text" spellcheck="false" autocomplete="off" class="model-filter" data-path="" />
                         <button id="reload-models" class="secondaryButton reloadModels"><i class='fa-solid fa-rotate'></i></button>
                         <a href="https://github.com/easydiffusion/easydiffusion/wiki/Custom-Models" target="_blank"><i class="fa-solid fa-circle-question help-btn"><span class="simple-tooltip top-left">Click to learn more about custom models</span></i></a>
                     </td></tr>
                     <tr class="pl-5 displayNone" id="enable_trt_config">
-                        <td><label for="convert_to_tensorrt">Convert to TensorRT:</label></td>
+                        <td><label for="convert_to_tensorrt">Enable TensorRT:</label></td>
                         <td class="diffusers-restart-needed">
                             <input id="convert_to_tensorrt" name="convert_to_tensorrt" type="checkbox">
-                            <a href="https://github.com/easydiffusion/easydiffusion/wiki/TensorRT" target="_blank"><i class="fa-solid fa-circle-question help-btn"><span class="simple-tooltip top-left">Click to learn more about TensorRT</span></i></a>
+                            <!-- <label><small>Takes upto 20 mins the first time</small></label> -->
-                            <label><small>Takes upto 20 mins the first time</small></label>
                         </td>
                     </tr>
                     <tr class="pl-5 displayNone" id="clip_skip_config">
@@ -165,6 +166,63 @@
                             <a href="https://github.com/easydiffusion/easydiffusion/wiki/Clip-Skip" target="_blank"><i class="fa-solid fa-circle-question help-btn"><span class="simple-tooltip top-left">Click to learn more about Clip Skip</span></i></a>
                         </td>
                     </tr>
+                    <tr id="controlnet_model_container" class="pl-5">
+                        <td><label for="controlnet_model">ControlNet Image:</label></td>
+                        <td class="diffusers-restart-needed">
+                            <div id="control_image_wrapper" class="preview_image_wrapper">
+                                <img id="control_image_preview" class="image_preview" src="" crossorigin="anonymous" />
+                                <span id="control_image_size_box" class="img_bottom_label"></span>
+                                <button class="control_image_clear image_clear_btn"><i class="fa-solid fa-xmark"></i></button>
+                            </div>
+                            <input id="control_image" name="control_image" type="file" />
+                            <a href="https://github.com/easydiffusion/easydiffusion/wiki/ControlNet" target="_blank"><i class="fa-solid fa-circle-question help-btn"><span class="simple-tooltip top-left">Click to learn more about ControlNets</span></i></a>
+                            <div id="controlnet_config" class="displayNone">
+                                <label><small>Filter to apply:</small></label>
+                                <select id="control_image_filter">
+                                    <option value="">None</option>
+                                    <optgroup label="Pose">
+                                        <option value="openpose">OpenPose (*)</option>
+                                        <option value="openpose_face">OpenPose face</option>
+                                        <option value="openpose_faceonly">OpenPose face-only</option>
+                                        <option value="openpose_hand">OpenPose hand</option>
+                                        <option value="openpose_full">OpenPose full</option>
+                                    </optgroup>
+                                    <optgroup label="Outline">
+                                        <option value="canny">Canny (*)</option>
+                                        <option value="mlsd">Straight lines</option>
+                                        <option value="scribble_hed">Scribble hed (*)</option>
+                                        <option value="scribble_hedsafe">Scribble hedsafe</option>
+                                        <option value="scribble_pidinet">Scribble pidinet</option>
+                                        <option value="scribble_pidsafe">Scribble pidsafe</option>
+                                        <option value="softedge_hed">Softedge hed</option>
+                                        <option value="softedge_hedsafe">Softedge hedsafe</option>
+                                        <option value="softedge_pidinet">Softedge pidinet</option>
+                                        <option value="softedge_pidsafe">Softedge pidsafe</option>
+                                    </optgroup>
+                                    <optgroup label="Depth">
+                                        <option value="normal_bae">Normal bae (*)</option>
+                                        <option value="depth_midas">Depth midas</option>
+                                        <option value="depth_zoe">Depth zoe</option>
+                                        <option value="depth_leres">Depth leres</option>
+                                        <option value="depth_leres++">Depth leres++</option>
+                                    </optgroup>
+                                    <optgroup label="Line art">
+                                        <option value="lineart_coarse">Lineart coarse</option>
+                                        <option value="lineart_realistic">Lineart realistic</option>
+                                        <option value="lineart_anime">Lineart anime</option>
+                                    </optgroup>
+                                    <optgroup label="Misc">
+                                        <option value="shuffle">Shuffle</option>
+                                        <option value="segment">Segment</option>
+                                    </optgroup>
+                                </select>
+                                <br/>
+                                <label for="controlnet_model"><small>Model:</small></label> <input id="controlnet_model" type="text" spellcheck="false" autocomplete="off" class="model-filter" data-path="" />
+                                <br/>
+                                <label><small>Will download the necessary models, the first time.</small></label>
+                            </div>
+                        </td>
+                    </tr>
                     <tr class="pl-5"><td><label for="vae_model">Custom VAE:</label></td><td>
                         <input id="vae_model" type="text" spellcheck="false" autocomplete="off" class="model-filter" data-path="" />
                         <a href="https://github.com/easydiffusion/easydiffusion/wiki/VAE-Variational-Auto-Encoder" target="_blank"><i class="fa-solid fa-circle-question help-btn"><span class="simple-tooltip top-left">Click to learn more about VAEs</span></i></a>
@@ -242,7 +300,7 @@
                         </select>
                         <label for="height"><small>(height)</small></label>
                         <div id="recent-resolutions-container">
-                            <span id="recent-resolutions-button" class="clickable"><i class="fa-solid fa-sliders"><span class="simple-tooltip top-left"> Recent sizes </span></i></span>
+                            <span id="recent-resolutions-button" class="clickable"><i class="fa-solid fa-sliders"><span class="simple-tooltip top-left"> Advanced sizes </span></i></span>
                             <div id="recent-resolutions-popup" class="displayNone">
                                 <small>Custom size:</small><br>
                                 <input id="custom-width" name="custom-width" type="number" min="128" value="512" onkeypress="preventNonNumericalInput(event)">
@@ -271,7 +329,7 @@
                             <button class="add_model_entry"><i class="fa-solid fa-plus"></i> add another LoRA</button>
                         </td>
                     </tr>
-                    <tr class="pl-5"><td><label for="hypernetwork_model">Hypernetwork:</label></td><td>
+                    <tr id="hypernetwork_model_container" class="pl-5"><td><label for="hypernetwork_model">Hypernetwork:</label></td><td>
                         <input id="hypernetwork_model" type="text" spellcheck="false" autocomplete="off" class="model-filter" data-path="" />
                     </td></tr>
                     <tr id="hypernetwork_strength_container" class="pl-5">
@@ -460,7 +518,9 @@
         </div>
         <div id="tab-content-gallery" class="tab-content">
             <button class="primaryButton" onclick="refreshGallery()">Refresh</button>
-            <div id="imagecontainer"></div>
+            <div class="gallery">
+                <div class="gallery-container" id="imagecontainer"></div>
+            </div>
         </div>
     </div>
     
@@ -630,6 +690,15 @@
             </button>
             <i class="fa-solid fa-magnifying-glass"></i>
             <input id="embeddings-search-box" type="text" spellcheck="false" autocomplete="off" placeholder="Search...">
+            <label for="embedding-card-size-selector"><small>Thumbnail Size:</small></label>
+            <select id="embedding-card-size-selector" name="embedding-card-size-selector">
+                <option value="-2">0</option>
+                <option value="-1">1</option>
+                <option value="0">2</option>
+                <option value="1" selected>3</option>
+                <option value="2">4</option>
+                <option value="3">5</option>
+            </select>
             <span style="float:right;"><label>Mode:</label>&nbsp;<select id="embeddings-mode"><option value="insert">Insert at cursor position</option><option value="append">Append at the end</option></select>
         </div>
         <div id="embeddings-list">
@@ -637,6 +706,34 @@
         </div>
     </dialog>
 
+    <dialog id="use-as-thumb-dialog">
+        <div id="use-as-thumb-dialog-header" class="dialog-header">
+            <div id="use-as-thumb-dialog-header-left" class="dialog-header-left">
+                <h4>Use as thumbnail</h4>
+                <span>Use a pictures as thumbnail for embeddings, LORAs, etc.</span>
+            </div>
+            <div id="use-as-thumb-dialog-header-right">
+                <i id="use-as-thumb-dialog-close-button" class="fa-solid fa-xmark fa-lg"></i>
+            </div>
+        </div>
+        <div>
+            <div class="use-as-thumb-grid">
+                <div class="use-as-thumb-preview">
+                    <div id="use-as-thumb-img-container"><img id="use-as-thumb-image" src="/media/images/noimg.png" width="512" height="512"></div>
+                </div>
+                <div class="use-as-thumb-select">
+                    <label for="use-as-thumb-select">Use the thumbnail for:</label><br>
+                    <select id="use-as-thumb-select" size="16" multiple>
+                    </select>
+                </div>
+                <div class="use-as-thumb-buttons">
+                    <button class="tertiaryButton" id="use-as-thumb-save">Save thumbnail</button>
+                    <button class="tertiaryButton" id="use-as-thumb-cancel">Cancel</button>
+                </div>
+            </div>
+        </div>
+    </dialog>
+
     <div id="image-editor" class="popup image-editor-popup">
         <div>
             <i class="close-button fa-solid fa-xmark"></i>
@@ -713,7 +810,7 @@ async function init() {
             ping: onPing
         }
     })
-    splashScreen()
+    // splashScreen()
 
     // load models again, but scan for malicious this time
     await getModels(true)

ui/main.py

@@ -1,11 +1,12 @@
 from easydiffusion import model_manager, app, server, bucket_manager
 from easydiffusion.server import server_api  # required for uvicorn
 
+app.init()
+
 server.init()
 
 # Init the app
 model_manager.init()
-app.init()
 app.init_render_threads()
 bucket_manager.init()
 

ui/media/css/croppr.css

@@ -0,0 +1,58 @@
+.croppr-container * {
+  user-select: none;
+  -moz-user-select: none;
+  -webkit-user-select: none;
+  -ms-user-select: none;
+  box-sizing: border-box;
+  -webkit-box-sizing: border-box;
+  -moz-box-sizing: border-box;
+}
+
+.croppr-container img {
+  vertical-align: middle;
+  max-width: 100%;
+}
+
+.croppr {
+  position: relative;
+  display: inline-block;
+}
+
+.croppr-overlay {
+  background: rgba(0,0,0,0.5);
+  position: absolute;
+  top: 0;
+  right: 0;
+  bottom: 0;
+  left: 0;
+  z-index: 1;
+  cursor: crosshair;
+}
+
+.croppr-region {
+  border: 1px dashed rgba(0, 0, 0, 0.5);
+  position: absolute;
+  z-index: 3;
+  cursor: move;
+  top: 0;
+}
+
+.croppr-imageClipped {
+  position: absolute;
+  top: 0;
+  right: 0;
+  bottom: 0;
+  left: 0;
+  z-index: 2;
+  pointer-events: none;
+}
+
+.croppr-handle {
+  border: 1px solid black;
+  background-color: white;
+  width: 10px;
+  height: 10px;
+  position: absolute;
+  z-index: 4;
+  top: 0;
+}

ui/media/css/main.css

@@ -794,7 +794,7 @@ div.img-preview img {
     margin-bottom: 8px;
 }
 
-#init_image_preview_container:not(.has-image) #init_image_wrapper,
+#init_image_preview_container:not(.has-image) .preview_image_wrapper,
 #init_image_preview_container:not(.has-image) #inpaint_button_container  {
     display: none;
 }
@@ -831,14 +831,14 @@ div.img-preview img {
     gap: 8px;
 }
 
-#init_image_wrapper {
+.preview_image_wrapper {
     grid-row: span 3;
     position: relative;
     width: fit-content;
     max-height: 150px;
 }
 
-#init_image_preview {
+.image_preview {
     max-height: 150px;
     height: 100%;
     width: 100%;
@@ -1650,6 +1650,35 @@ body.wait-pause {
     }
 }
 
+.spinner-container {
+    width: 80px;
+    height: 100px;
+    margin: 100px auto;
+    margin-top: 30vH;
+}
+
+.spinner-block {
+    position: relative;
+    box-sizing: border-box;
+    float: left;
+    margin: 0 10px 10px 0;
+    width: 12px;
+    height: 12px;
+    border-radius: 3px;
+    background: var(--accent-color);
+}
+
+.spinner-block:nth-child(4n+1) { animation: spinner-wave 2s ease .0s infinite; }
+.spinner-block:nth-child(4n+2) { animation: spinner-wave 2s ease .2s infinite; }
+.spinner-block:nth-child(4n+3) { animation: spinner-wave 2s ease .4s infinite; }
+.spinner-block:nth-child(4n+4) { animation: spinner-wave 2s ease .6s infinite; margin-right: 0; }
+
+@keyframes spinner-wave {
+    0%   { top: 0;     opacity: 1; }
+    50%  { top: 30px;  opacity: .2; }
+    100% { top: 0;     opacity: 1; }
+}
+
 #embeddings-dialog {
     overflow: clip;
 }
@@ -1741,6 +1770,32 @@ body.wait-pause {
     float: right;
 }
 
+.use-as-thumb-grid {  display: grid;
+  grid-template-columns: 1fr auto;
+  grid-template-rows: 1fr auto;
+  gap: 8px 8px;
+  grid-auto-flow: row;
+  grid-template-areas:
+    "uat-preview uat-select"
+    "uat-preview uat-buttons";
+}
+
+.use-as-thumb-preview {
+  justify-self: center;
+  align-self: center;
+  grid-area: uat-preview;
+}
+
+.use-as-thumb-select { 
+    grid-area: uat-select;
+}
+
+.use-as-thumb-buttons {
+  justify-self: center;
+  grid-area: uat-buttons;
+}
+
+
 .diffusers-disabled-on-startup .diffusers-restart-needed {
     font-size: 0;
 }
@@ -1818,23 +1873,57 @@ div#enlarge-buttons {
     font-size: 10pt;
 }
 
-/* Gallery CSS */
+#controlnet_model_container small {
-#imagecontainer {
+    color: var(--text-color)
-    display: flex;
+}
-    justify-content: space-around;
+#control_image {
-    flex-flow: row wrap;
+    width: 130pt;
-    align-items: center;
+}
+#controlnet_model {
+    width: 77%;
 }
 
-#imagecontainer>img {
+/* hack for fixing Image Modifier Improvements plugin */
-    width: 30vw;
+#imageTagPopupContainer {
-    min-width: 256px;
+    position: absolute;
-    max-width: 1024px;
-    height: auto;
-    margin-block: 1vh;
-    border: 4px white solid;
 }
 
+/* Gallery CSS */
+.gallery {
+  display: flex;
+  justify-content: center;
+  align-items: center;
+  flex-direction: column;
+  font-family: sans-serif;
+}
+
+.gallery-container {
+  columns: 5 ;
+  column-gap: 1.5rem;
+  width: 95%;
+  margin: 0 0 ;
+}
+.gallery-container  div {
+    margin: 0 1.5rem 1.5rem 0;
+    display: inline-block;
+    width: 100%;
+    padding: 5px;
+    
+    transition: all .75s ease-in-out;
+}
+
+.gallery-container div img {
+    width: 100%;
+    border-radius: 5px;
+    transition: all .25s ease-in-out;
+    box-shadow: 5px 5px 5px rgba(0,0,0,0.4);
+}
+
+.gallery-container div img:hover {
+    box-shadow: 1px 1px 15px rgba(32,0,128,0.8);
+}
+
+
 #tab-content-gallery>button {
     margin: 8px;
 }

ui/media/js/auto-save.js

@@ -45,6 +45,7 @@ const SETTINGS_IDS_LIST = [
     "sound_toggle",
     "vram_usage_level",
     "confirm_dangerous_actions",
+    "profileName",
     "metadata_output_format",
     "auto_save_settings",
     "apply_color_correction",
@@ -54,6 +55,8 @@ const SETTINGS_IDS_LIST = [
     "zip_toggle",
     "tree_toggle",
     "json_toggle",
+    "extract_lora_from_prompt",
+    "embedding-card-size-selector",
 ]
 
 const IGNORE_BY_DEFAULT = ["prompt"]

ui/media/js/main.js

@@ -93,6 +93,11 @@ let initImagePreview = document.querySelector("#init_image_preview")
 let initImageSizeBox = document.querySelector("#init_image_size_box")
 let maskImageSelector = document.querySelector("#mask")
 let maskImagePreview = document.querySelector("#mask_preview")
+let controlImageSelector = document.querySelector("#control_image")
+let controlImagePreview = document.querySelector("#control_image_preview")
+let controlImageClearBtn = document.querySelector(".control_image_clear")
+let controlImageContainer = document.querySelector("#control_image_wrapper")
+let controlImageFilterField = document.querySelector("#control_image_filter")
 let applyColorCorrectionField = document.querySelector("#apply_color_correction")
 let strictMaskBorderField = document.querySelector("#strict_mask_border")
 let colorCorrectionSetting = document.querySelector("#apply_color_correction_setting")
@@ -114,6 +119,7 @@ let codeformerFidelityField = document.querySelector("#codeformer_fidelity")
 let stableDiffusionModelField = new ModelDropdown(document.querySelector("#stable_diffusion_model"), "stable-diffusion")
 let clipSkipField = document.querySelector("#clip_skip")
 let tilingField = document.querySelector("#tiling")
+let controlnetModelField = new ModelDropdown(document.querySelector("#controlnet_model"), "controlnet", "None", false)
 let vaeModelField = new ModelDropdown(document.querySelector("#vae_model"), "vae", "None")
 let hypernetworkModelField = new ModelDropdown(document.querySelector("#hypernetwork_model"), "hypernetwork", "None")
 let hypernetworkStrengthSlider = document.querySelector("#hypernetwork_strength_slider")
@@ -135,6 +141,7 @@ let embeddingsDialogCloseBtn = embeddingsDialog.querySelector("#embeddings-dialo
 let embeddingsSearchBox = document.querySelector("#embeddings-search-box")
 let embeddingsList = document.querySelector("#embeddings-list")
 let embeddingsModeField = document.querySelector("#embeddings-mode")
+let embeddingsCardSizeSelector = document.querySelector("#embedding-card-size-selector")
 
 let positiveEmbeddingText = document.querySelector("#positive-embedding-text")
 let negativeEmbeddingText = document.querySelector("#negative-embedding-text")
@@ -164,6 +171,12 @@ let saveAllTreeToggle = document.querySelector("#tree_toggle")
 let saveAllJSONToggle = document.querySelector("#json_toggle")
 let saveAllFoldersOption = document.querySelector("#download-add-folders")
 let splashScreenPopup = document.querySelector("#splash-screen")
+let useAsThumbDialog = document.querySelector("#use-as-thumb-dialog")
+let useAsThumbDialogCloseBtn = document.querySelector("#use-as-thumb-dialog-close-button")
+let useAsThumbImageContainer = document.querySelector("#use-as-thumb-img-container")
+let useAsThumbSelect = document.querySelector("#use-as-thumb-select")
+let useAsThumbSaveBtn = document.querySelector("#use-as-thumb-save")
+let useAsThumbCancelBtn = document.querySelector("#use-as-thumb-cancel")
 
 let maskSetting = document.querySelector("#enable_mask")
 
@@ -177,6 +190,8 @@ let undoBuffer = []
 const UNDO_LIMIT = 20
 const MAX_IMG_UNDO_ENTRIES = 5
 
+let IMAGE_STEP_SIZE = 64
+
 let loraModels = []
 
 imagePreview.addEventListener("drop", function(ev) {
@@ -670,19 +685,126 @@ function onMakeSimilarClick(req, img) {
     createTask(newTaskRequest)
 }
 
+// gets a flat list of all models of a certain type, ignoring directories
+function getAllModelNames(type) {
+    function f(tree) {
+        if (tree == undefined) {
+            return []
+        }
+        let result=[]; 
+        tree.forEach( e => { 
+            if (typeof(e) == "object") {
+                result = result.concat( f(e[1]))
+            } else {
+                result.push(e) 
+            }
+        });
+        return result
+    }
+    return f(modelsOptions[type])
+}
+
 function onUseAsThumbnailClick(req, img) {
-    console.log(req)
+    let scale = 1
-    console.log(img)
+    let targetWidth = img.naturalWidth
-    let embedding = prompt("Embedding name")
+    let targetHeight = img.naturalHeight
-    fetch(img.src)
+    let resize = false
+    onUseAsThumbnailClick.img = img
+
+    if ( typeof(onUseAsThumbnailClick.croppr) == 'undefined' ) {
+        onUseAsThumbnailClick.croppr = new Croppr("#use-as-thumb-image", { aspectRatio: 1, minSize: [384,384,'px'], startSize:  [512, 512, 'px'], returnMode:"real" })
+    }
+
+    if (img.naturalWidth > img.naturalHeight) {
+        if (img.naturalWidth > 768) {
+           scale = 768 / img.naturalWidth
+           targetWidth  = 768
+           targetHeight = (img.naturalHeight*scale)>>>0
+           resize = true
+        }
+    } else {
+        if (img.naturalHeight > 768) {
+            scale = 768 / img.naturalHeight
+            targetHeight = 768
+            targetWidth  = (img.naturalWidth*scale)>>>0
+            resize = true
+        }
+    }
+
+    onUseAsThumbnailClick.croppr.options.minSize = {width: 384*scale>>>0, height: 384*scale>>>0}
+    onUseAsThumbnailClick.croppr.options.startSize = {width: 512*scale>>>0, height: 512*scale>>>0}
+
+    if (resize) {
+        const canvas = document.createElement('canvas')
+        canvas.width = targetWidth
+        canvas.height = targetHeight
+        const ctx = canvas.getContext('2d')
+        ctx.drawImage(img, 0, 0, targetWidth, targetHeight)
+
+        onUseAsThumbnailClick.croppr.setImage(canvas.toDataURL('image/png'))
+    } else {
+        onUseAsThumbnailClick.croppr.setImage(img.src)
+    }
+
+    let embeddings = getAllModelNames("embeddings").filter( e => req.prompt.includes(e) || req.negative_prompt.includes(e) )
+    let LORA = []
+
+    if ("use_lora_model" in req) {
+        LORA=req.use_lora_model
+    }
+
+    let optgroup = document.createElement("optgroup")
+    optgroup.label = "Embeddings"
+    optgroup.replaceChildren(...embeddings.map(e => { 
+        let option = document.createElement("option")
+        option.innerText = e
+        option.dataset["type"] = "embeddings"
+        return option
+    }))
+    
+    useAsThumbSelect.replaceChildren(optgroup)
+    useAsThumbDialog.showModal()
+    onUseAsThumbnailClick.scale = scale
+}
+
+modalDialogCloseOnBackdropClick(useAsThumbDialog)
+makeDialogDraggable(useAsThumbDialog)
+
+useAsThumbDialogCloseBtn.addEventListener("click", () => {
+    useAsThumbDialog.close()
+})
+
+useAsThumbCancelBtn.addEventListener("click", () => {
+    useAsThumbDialog.close()
+})
+
+useAsThumbSaveBtn.addEventListener("click", (e) => {
+    let scale = 1/onUseAsThumbnailClick.scale
+    let crop = onUseAsThumbnailClick.croppr.getValue()
+
+    let len = Math.max(crop.width*scale, 384)
+    let profileName = profileNameField.value
+
+    cropImageDataUrl(onUseAsThumbnailClick.img.src, crop.x*scale, crop.y*scale, len, len)
+        .then(thumb => fetch(thumb))
         .then(response => response.blob())
         .then(async function(blob) {
             const formData  = new FormData()
             formData.append("file", blob)
-          const response = await fetch(`bucket/embeddings/${embedding}.jpg`, { method: 'POST', body: formData });
+            let options = useAsThumbSelect.selectedOptions
-          console.log(response)
+            let promises = []
+            for (let embedding of options) {
+                promises.push(fetch(`bucket/${profileName}/${embedding.dataset["type"]}/${embedding.value}.png`, { method: 'POST', body: formData }))
+            }
+            return Promise.all(promises)
+        }).then(() => {
+            useAsThumbDialog.close()
         })
-}
+        .catch(error => {
+            console.error(error)
+            showToast("Couldn't save thumbnail.<br>"+error)
+        })
+})
 
 function enqueueImageVariationTask(req, img, reqDiff) {
     const imageSeed = img.getAttribute("data-seed")
@@ -1361,9 +1483,25 @@ function createTask(task) {
 
 function getCurrentUserRequest() {
     const numOutputsTotal = parseInt(numOutputsTotalField.value)
-    const numOutputsParallel = parseInt(numOutputsParallelField.value)
+    let numOutputsParallel = parseInt(numOutputsParallelField.value)
     const seed = randomSeedField.checked ? Math.floor(Math.random() * (2 ** 32 - 1)) : parseInt(seedField.value)
 
+    // if (
+    //     testDiffusers.checked &&
+    //     document.getElementById("toggle-tensorrt-install").innerHTML == "Uninstall" &&
+    //     document.querySelector("#convert_to_tensorrt").checked
+    // ) {
+    //     // TRT enabled
+
+    //     numOutputsParallel = 1 // force 1 parallel
+    // }
+
+    // clamp to multiple of 8
+    let width = parseInt(widthField.value)
+    let height = parseInt(heightField.value)
+    width = width - (width % IMAGE_STEP_SIZE)
+    height = height - (height % IMAGE_STEP_SIZE)
+
     const newTask = {
         batchesDone: 0,
         numOutputsTotal: numOutputsTotal,
@@ -1376,8 +1514,8 @@ function getCurrentUserRequest() {
             num_outputs: numOutputsParallel,
             num_inference_steps: parseInt(numInferenceStepsField.value),
             guidance_scale: parseFloat(guidanceScaleField.value),
-            width: parseInt(widthField.value),
+            width: width,
-            height: parseInt(heightField.value),
+            height: height,
             // allow_nsfw: allowNSFWField.checked,
             vram_usage_level: vramUsageLevelField.value,
             sampler_name: samplerField.value,
@@ -1451,6 +1589,29 @@ function getCurrentUserRequest() {
     if (testDiffusers.checked && document.getElementById("toggle-tensorrt-install").innerHTML == "Uninstall") {
         // TRT is installed
         newTask.reqBody.convert_to_tensorrt = document.querySelector("#convert_to_tensorrt").checked
+        let trtBuildConfig = {
+            batch_size_range: [
+                parseInt(document.querySelector("#trt-build-min-batch").value),
+                parseInt(document.querySelector("#trt-build-max-batch").value),
+            ],
+            dimensions_range: [],
+        }
+
+        let sizes = [512, 768, 1024, 1280, 1536]
+        sizes.forEach((i) => {
+            let el = document.querySelector("#trt-build-res-" + i)
+            if (el.checked) {
+                trtBuildConfig["dimensions_range"].push([i, i + 256])
+            }
+        })
+        newTask.reqBody.trt_build_config = trtBuildConfig
+    }
+    if (controlnetModelField.value !== "" && IMAGE_REGEX.test(controlImagePreview.src)) {
+        newTask.reqBody.use_controlnet_model = controlnetModelField.value
+        newTask.reqBody.control_image = controlImagePreview.src
+        if (controlImageFilterField.value !== "") {
+            newTask.reqBody.control_filter_to_apply = controlImageFilterField.value
+        }
     }
 
     return newTask
@@ -1858,6 +2019,51 @@ function onFixFaceModelChange() {
 gfpganModelField.addEventListener("change", onFixFaceModelChange)
 onFixFaceModelChange()
 
+function onControlnetModelChange() {
+    let configBox = document.querySelector("#controlnet_config")
+    if (IMAGE_REGEX.test(controlImagePreview.src)) {
+        configBox.classList.remove("displayNone")
+        controlImageContainer.classList.remove("displayNone")
+    } else {
+        configBox.classList.add("displayNone")
+        controlImageContainer.classList.add("displayNone")
+    }
+}
+controlImagePreview.addEventListener("load", onControlnetModelChange)
+controlImagePreview.addEventListener("unload", onControlnetModelChange)
+onControlnetModelChange()
+
+function onControlImageFilterChange() {
+    let filterId = controlImageFilterField.value
+    if (filterId.includes("openpose")) {
+        controlnetModelField.value = "control_v11p_sd15_openpose"
+    } else if (filterId === "canny") {
+        controlnetModelField.value = "control_v11p_sd15_canny"
+    } else if (filterId === "mlsd") {
+        controlnetModelField.value = "control_v11p_sd15_mlsd"
+    } else if (filterId === "mlsd") {
+        controlnetModelField.value = "control_v11p_sd15_mlsd"
+    } else if (filterId.includes("scribble")) {
+        controlnetModelField.value = "control_v11p_sd15_scribble"
+    } else if (filterId.includes("softedge")) {
+        controlnetModelField.value = "control_v11p_sd15_softedge"
+    } else if (filterId === "normal_bae") {
+        controlnetModelField.value = "control_v11p_sd15_normalbae"
+    } else if (filterId.includes("depth")) {
+        controlnetModelField.value = "control_v11f1p_sd15_depth"
+    } else if (filterId === "lineart_anime") {
+        controlnetModelField.value = "control_v11p_sd15s2_lineart_anime"
+    } else if (filterId.includes("lineart")) {
+        controlnetModelField.value = "control_v11p_sd15_lineart"
+    } else if (filterId === "shuffle") {
+        controlnetModelField.value = "control_v11e_sd15_shuffle"
+    } else if (filterId === "segment") {
+        controlnetModelField.value = "control_v11p_sd15_seg"
+    }
+}
+controlImageFilterField.addEventListener("change", onControlImageFilterChange)
+onControlImageFilterChange()
+
 upscaleModelField.disabled = !useUpscalingField.checked
 upscaleAmountField.disabled = !useUpscalingField.checked
 useUpscalingField.addEventListener("change", function(e) {
@@ -2087,6 +2293,7 @@ function checkRandomSeed() {
 randomSeedField.addEventListener("input", checkRandomSeed)
 checkRandomSeed()
 
+// warning: the core plugin `image-editor-improvements.js:172` replaces loadImg2ImgFromFile() with a custom version
 function loadImg2ImgFromFile() {
     if (initImageSelector.files.length === 0) {
         return
@@ -2148,6 +2355,47 @@ promptsFromFileBtn.addEventListener("click", function() {
     promptsFromFileSelector.click()
 })
 
+function loadControlnetImageFromFile() {
+    if (controlImageSelector.files.length === 0) {
+        return
+    }
+
+    let reader = new FileReader()
+    let file = controlImageSelector.files[0]
+
+    reader.addEventListener("load", function(event) {
+        controlImagePreview.src = reader.result
+    })
+
+    if (file) {
+        reader.readAsDataURL(file)
+    }
+}
+controlImageSelector.addEventListener("change", loadControlnetImageFromFile)
+
+function controlImageLoad() {
+    let w = controlImagePreview.naturalWidth
+    let h = controlImagePreview.naturalHeight
+    w = w - (w % IMAGE_STEP_SIZE)
+    h = h - (h % IMAGE_STEP_SIZE)
+
+    addImageSizeOption(w)
+    addImageSizeOption(h)
+
+    widthField.value = w
+    heightField.value = h
+    widthField.dispatchEvent(new Event("change"))
+    heightField.dispatchEvent(new Event("change"))
+}
+controlImagePreview.addEventListener("load", controlImageLoad)
+
+function controlImageUnload() {
+    controlImageSelector.value = null
+    controlImagePreview.src = ""
+    controlImagePreview.dispatchEvent(new Event("unload"))
+}
+controlImageClearBtn.addEventListener("click", controlImageUnload)
+
 promptsFromFileSelector.addEventListener("change", async function() {
     if (promptsFromFileSelector.files.length === 0) {
         return
@@ -2276,6 +2524,8 @@ function tunnelUpdate(event) {
     }
 }
 
+let trtSettingsForced = false
+
 function packagesUpdate(event) {
     let trtBtn = document.getElementById("toggle-tensorrt-install")
     let trtInstalled = "packages_installed" in event && "tensorrt" in event["packages_installed"]
@@ -2290,6 +2540,23 @@ function packagesUpdate(event) {
 
     if (document.getElementById("toggle-tensorrt-install").innerHTML == "Uninstall") {
         document.querySelector("#enable_trt_config").classList.remove("displayNone")
+        document.querySelector("#trt-build-config").classList.remove("displayNone")
+
+        if (!trtSettingsForced) {
+            // settings for demo
+            promptField.value = "Dragons fighting with a knight, castle, war scene, fantasy, cartoon, flames, HD"
+            seedField.value = 3187947173
+            widthField.value = 1024
+            heightField.value = 768
+            randomSeedField.checked = false
+            seedField.disabled = false
+            stableDiffusionModelField.value = "sd-v1-4"
+
+            // numOutputsParallelField.classList.add("displayNone")
+            // document.querySelector("#num_outputs_parallel_label").classList.add("displayNone")
+
+            trtSettingsForced = true
+        }
     }
 }
 
@@ -2371,40 +2638,53 @@ document.getElementById("toggle-tensorrt-install").addEventListener("click", fun
 
 /* Embeddings */
 
-let icl = []
 function updateEmbeddingsList(filter = "") {
     function html(model, iconlist = [], prefix = "", filter = "") {
         filter = filter.toLowerCase()
-        let toplevel = ""
+        let toplevel = document.createElement("div")
-        let folders = ""
+        let folders = document.createElement("div")
-        console.log(iconlist)
         let embIcon = Object.assign({}, ...iconlist.map( x=> ({[x.toLowerCase().split('.').slice(0,-1).join('.')]:x})))
 
+        let profileName = profileNameField.value
         model?.forEach((m) => {
             if (typeof m == "string") {
                 let token=m.toLowerCase()
                 if (token.search(filter) != -1) {
+                    let button
+                    if (iconlist.length==0) {
+                        button = document.createElement("button")
+                        button.innerText="m"
+                    } else {
                         let img = '/media/images/noimg.png'
                         if (token in embIcon) {
-                        img = `/bucket/embeddings/${embIcon[token]}`
+                            img = `/bucket/${profileName}/embeddings/${embIcon[token]}`
                         }
-                    toplevel += `<button data-embedding="${m}"><img src="${img}" height="128" width="128"><br>${m}</button> `
+                        button = createModifierCard(m, [img,img], true)
+                    }
+                    button.dataset["embedding"] = m
+                    button.addEventListener("click", onButtonClick)
+                    toplevel.appendChild(button)
                 }
             } else {
                 let subdir = html(m[1], iconlist, prefix + m[0] + "/", filter)
-                if (subdir != "") {
+                if (typeof(subdir) == "object") {
-                    folders +=
+                    let div1 = document.createElement("div")
-                        `<div class="embedding-category"><h4 class="collapsible">${prefix}${m[0]}</h4><div class="collapsible-content">` +
+                    let div2 = document.createElement("div")
-                        subdir +
+                    div1.classList.add("collapsible-content")
-                        "</div></div>"
+                    div1.classList.add("embedding-category")
+                    div1.appendChild(subdir)
+                    div2.replaceChildren(htmlToElement(`<h4 class="collapsible">${prefix}${m[0]}</h4>`), div1)
+                    folders.appendChild(div2)
                 }
             }
         })
-        return toplevel + folders
+        let result = document.createElement("div")
+        result.replaceChildren(toplevel, htmlToElement('<br style="clear: both;">'), folders)
+        return result
     }
 
     function onButtonClick(e) {
-        let text = e.target.closest("button").dataset["embedding"]
+        let text = e.target.closest("[data-embedding]").dataset["embedding"]
         const insertIntoNegative = e.shiftKey || positiveEmbeddingText.classList.contains("displayNone")
 
         if (embeddingsModeField.value == "insert") {
@@ -2429,8 +2709,18 @@ function updateEmbeddingsList(filter = "") {
         }
     }
 
+    // Usually the rendering of the Embeddings HTML takes less than a second. In case it takes longer, show a spinner
+    embeddingsList.innerHTML = `
+        <div class="spinner-container">
+          <div class="spinner-block"></div> <div class="spinner-block"></div> <div class="spinner-block"></div> <div class="spinner-block"></div>
+          <div class="spinner-block"></div> <div class="spinner-block"></div> <div class="spinner-block"></div> <div class="spinner-block"></div>
+          <div class="spinner-block"></div> <div class="spinner-block"></div> <div class="spinner-block"></div> <div class="spinner-block"></div>
+          <div class="spinner-block"></div> <div class="spinner-block"></div> <div class="spinner-block"></div> <div class="spinner-block"></div>
+        </div>
+    `
+
     // Remove after fixing https://github.com/huggingface/diffusers/issues/3922
-    let warning = ""
+    let warning = "<div></div>"
     if (vramUsageLevelField.value == "low") {
         warning = `
             <div style="border-color: var(--accent-color); border-width: 4px; border-radius: 1em; border-style: solid; background: black; text-align: center; padding: 1em; margin: 1em; ">
@@ -2439,17 +2729,17 @@ function updateEmbeddingsList(filter = "") {
     }
     // END of remove block
 
-    fetch("/bucket/embeddings/")
+    let profileName = profileNameField.value
-       .then(response => response.json())
+    fetch(`/bucket/${profileName}/embeddings/`)
-       .then(iconlist => {
+       .then(response => response.status==200 ? response.json(): [])
-           embeddingsList.innerHTML = warning + html(modelsOptions.embeddings, iconlist, "", filter)
+       .then(async function(iconlist) {
-           embeddingsList.querySelectorAll("button").forEach((b) => {
+
-               b.addEventListener("click", onButtonClick)
+           embeddingsList.replaceChildren(htmlToElement(warning), html(modelsOptions.embeddings, iconlist, "", filter))
-           })
            createCollapsibles(embeddingsList)
            if (filter != "") {
                embeddingsExpandAll()
            }
+           resizeModifierCards(embeddingsCardSizeSelector.value)
        })
 }
 
@@ -2458,23 +2748,33 @@ function showEmbeddingDialog() {
     embeddingsSearchBox.value = ""
     embeddingsDialog.showModal()
 }
+
 embeddingsButton.addEventListener("click", () => {
     positiveEmbeddingText.classList.remove("displayNone")
     negativeEmbeddingText.classList.add("displayNone")
     showEmbeddingDialog()
 })
+
 negativeEmbeddingsButton.addEventListener("click", () => {
     positiveEmbeddingText.classList.add("displayNone")
     negativeEmbeddingText.classList.remove("displayNone")
     showEmbeddingDialog()
 })
+
 embeddingsDialogCloseBtn.addEventListener("click", (e) => {
     embeddingsDialog.close()
 })
+
 embeddingsSearchBox.addEventListener("input", (e) => {
     updateEmbeddingsList(embeddingsSearchBox.value)
 })
 
+embeddingsCardSizeSelector.addEventListener("change", (e) => {
+    resizeModifierCards(embeddingsCardSizeSelector.value)
+})
+
+
+
 modalDialogCloseOnBackdropClick(embeddingsDialog)
 makeDialogDraggable(embeddingsDialog)
 
@@ -2522,10 +2822,6 @@ embeddingsCollapsiblesBtn.addEventListener("click", (e) => {
     }
 })
 
-if (testDiffusers.checked) {
-    document.getElementById("embeddings-container").classList.remove("displayNone")
-}
-
 /* Pause function */
 document.querySelectorAll(".tab").forEach(linkTabContents)
 
@@ -2789,13 +3085,25 @@ let recentResolutionsValues = []
 })()
 
 /* Gallery JS */
+function galleryImage(item) {
+    let div = document.createElement("div")
+    let img = document.createElement("img")
+
+    img.src = "/image/" + item.path
+    img.dataset["request"] = JSON.stringify(item)
+    div.appendChild(img)
+    return div
+}
 
 function refreshGallery() {
     let container = document.getElementById("imagecontainer")
-    container.remove()
+    container.innerHTML=""
     fetch('/all_images')
-        .then(response => response.text())
+        .then(response => response.json())
-        .then(text => new DOMParser().parseFromString(text, 'text/html'))
+        .then(json => {
-        .then(html_like => html_like.getElementsByTagName('div')[0])
+            console.log(json)
-        .then(div => document.getElementById("tab-content-gallery").appendChild(div))
+            json.forEach( item => {
+                container.appendChild(galleryImage(item))
+            })
+         })
 }

ui/media/js/parameters.js

@@ -121,6 +121,15 @@ var PARAMETERS = [
         icon: "fa-arrow-down-short-wide",
         default: false,
     },
+    {
+        id: "extract_lora_from_prompt",
+        type: ParameterType.checkbox,
+        label: "Extract LoRA tags from the prompt",
+        note:
+            "Automatically extract lora tags like <lora:name:0.4> from the prompt, and apply the correct LoRA (if present)",
+        icon: "fa-code",
+        default: true,
+    },
     {
         id: "ui_open_browser_on_start",
         type: ParameterType.checkbox,
@@ -185,6 +194,16 @@ var PARAMETERS = [
         icon: "fa-check-double",
         default: true,
     },
+    {
+        id: "profileName",
+        type: ParameterType.custom,
+        label: "Profile Name",
+        note: "Name of the profile for model manager settings, e.g. thumbnails for embeddings. Use this to have different settings for different users.",
+        render: (parameter) => {
+            return `<input id="${parameter.id}" name="${parameter.id}" value="default" size="12">`
+        },
+        icon: "fa-user-gear",
+    },
     {
         id: "listen_to_network",
         type: ParameterType.checkbox,
@@ -220,11 +239,11 @@ var PARAMETERS = [
     {
         id: "test_diffusers",
         type: ParameterType.checkbox,
-        label: "Test Diffusers",
+        label: "Use the new v3 engine (diffusers)",
         note:
-            "<b>Experimental! Can have bugs!</b> Use upcoming features (like LoRA) in our new engine. Please press Save, then restart the program after changing this.",
+            "Use our new v3 engine, with additional features like LoRA, ControlNet, SDXL, Embeddings, Tiling and lots more! Please press Save, then restart the program after changing this.",
         icon: "fa-bolt",
-        default: false,
+        default: true,
         saveInAppConfig: true,
     },
     {
@@ -248,7 +267,19 @@ var PARAMETERS = [
         label: "NVIDIA TensorRT",
         note: `Faster image generation by converting your Stable Diffusion models to the NVIDIA TensorRT format. You can choose the
                models to convert. Download size: approximately 2 GB.<br/><br/>
-               <b>Early access version:</b> support for LoRA is still under development.`,
+               <b>Early access version:</b> support for LoRA is still under development.
+               <div id="trt-build-config" class="displayNone">
+                    <h3>Build Config:</h3>
+                    Batch size range:
+                    <label>Min:</label> <input id="trt-build-min-batch" type="number" min="1" value="1" style="width: 40pt" />
+                    <label>Max:</label> <input id="trt-build-max-batch" type="number" min="1" value="1" style="width: 40pt" /><br/><br/>
+                    <b>Build for resolutions</b>:<br/>
+                    <input id="trt-build-res-512" type="checkbox" value="1" /> 512x512 to 768x768<br/>
+                    <input id="trt-build-res-768" type="checkbox" value="1" checked /> 768x768 to 1024x1024<br/>
+                    <input id="trt-build-res-1024" type="checkbox" value="1" /> 1024x1024 to 1280x1280<br/>
+                    <input id="trt-build-res-1280" type="checkbox" value="1" /> 1280x1280 to 1536x1536<br/>
+                    <input id="trt-build-res-1536" type="checkbox" value="1" /> 1536x1536 to 1792x1792<br/>
+               </div>`,
         icon: "fa-angles-up",
         render: () => '<button id="toggle-tensorrt-install" class="primaryButton">Install</button>',
         table: installExtrasTable,
@@ -389,6 +420,7 @@ let useBetaChannelField = document.querySelector("#use_beta_channel")
 let uiOpenBrowserOnStartField = document.querySelector("#ui_open_browser_on_start")
 let confirmDangerousActionsField = document.querySelector("#confirm_dangerous_actions")
 let testDiffusers = document.querySelector("#test_diffusers")
+let profileNameField = document.querySelector("#profileName")
 
 let saveSettingsBtn = document.querySelector("#save-system-settings-btn")
 
@@ -433,7 +465,10 @@ async function getAppConfig() {
             listenPortField.value = config.net.listen_port
         }
 
-        const testDiffusersEnabled = config.test_diffusers && config.update_branch !== "main"
+        let testDiffusersEnabled = config.update_branch !== "main"
+        if (config.test_diffusers === false) {
+            testDiffusersEnabled = false
+        }
         testDiffusers.checked = testDiffusersEnabled
 
         if (config.config_on_startup) {
@@ -449,15 +484,22 @@ async function getAppConfig() {
         if (!testDiffusersEnabled) {
             document.querySelector("#lora_model_container").style.display = "none"
             document.querySelector("#tiling_container").style.display = "none"
+            document.querySelector("#controlnet_model_container").style.display = "none"
+            document.querySelector("#hypernetwork_model_container").style.display = ""
+            document.querySelector("#hypernetwork_strength_container").style.display = ""
 
             document.querySelectorAll("#sampler_name option.diffusers-only").forEach((option) => {
                 option.style.display = "none"
             })
-            customWidthField.step=64
+            IMAGE_STEP_SIZE = 64
-            customHeightField.step=64
+            customWidthField.step = IMAGE_STEP_SIZE
+            customHeightField.step = IMAGE_STEP_SIZE
         } else {
             document.querySelector("#lora_model_container").style.display = ""
             document.querySelector("#tiling_container").style.display = ""
+            document.querySelector("#controlnet_model_container").style.display = ""
+            document.querySelector("#hypernetwork_model_container").style.display = "none"
+            document.querySelector("#hypernetwork_strength_container").style.display = "none"
 
             document.querySelectorAll("#sampler_name option.k_diffusion-only").forEach((option) => {
                 option.style.display = "none"
@@ -465,8 +507,9 @@ async function getAppConfig() {
             document.querySelector("#clip_skip_config").classList.remove("displayNone")
             document.querySelector("#embeddings-button").classList.remove("displayNone")
             document.querySelector("#negative-embeddings-button").classList.remove("displayNone")
-            customWidthField.step=8
+            IMAGE_STEP_SIZE = 8
-            customHeightField.step=8
+            customWidthField.step = IMAGE_STEP_SIZE
+            customHeightField.step = IMAGE_STEP_SIZE
         }
 
         console.log("get config status response", config)

ui/media/js/searchable-models.js

@@ -552,17 +552,23 @@ class ModelDropdown {
                     this.createModelNodeList(`${folderName || ""}/${childFolderName}`, childModels, false)
                 )
             } else {
+                let modelId = model
+                let modelName = model
+                if (typeof model === "object") {
+                    modelId = Object.keys(model)[0]
+                    modelName = model[modelId]
+                }
                 const classes = ["model-file"]
                 if (isRootFolder) {
                     classes.push("in-root-folder")
                 }
                 // Remove the leading slash from the model path
-                const fullPath = folderName ? `${folderName.substring(1)}/${model}` : model
+                const fullPath = folderName ? `${folderName.substring(1)}/${modelId}` : modelId
                 modelsMap.set(
-                    model,
+                    modelId,
                     createElement("li", { "data-path": fullPath }, classes, [
                         createElement("i", undefined, ["fa-regular", "fa-file", "icon"]),
-                        model,
+                        modelName,
                     ])
                 )
             }
@@ -643,22 +649,6 @@ async function getModels(scanForMalicious = true) {
             makeImageBtn.disabled = true
         }
 
-        /* This code should no longer be needed. Commenting out for now, will cleanup later.
-        const sd_model_setting_key = "stable_diffusion_model"
-        const vae_model_setting_key = "vae_model"
-        const hypernetwork_model_key = "hypernetwork_model"
-
-        const stableDiffusionOptions = modelsOptions['stable-diffusion']
-        const vaeOptions = modelsOptions['vae']
-        const hypernetworkOptions = modelsOptions['hypernetwork']
-
-        // TODO: set default for model here too
-        SETTINGS[sd_model_setting_key].default = stableDiffusionOptions[0]
-        if (getSetting(sd_model_setting_key) == '' || SETTINGS[sd_model_setting_key].value == '') {
-            setSetting(sd_model_setting_key, stableDiffusionOptions[0])
-        }
-        */
-
         // notify ModelDropdown objects to refresh
         document.dispatchEvent(new Event("refreshModels"))
     } catch (e) {
@@ -667,4 +657,7 @@ async function getModels(scanForMalicious = true) {
 }
 
 // reload models button
-document.querySelector("#reload-models").addEventListener("click", () => getModels())
+document.querySelector("#reload-models").addEventListener("click", (e) => {
+    e.stopPropagation()
+    getModels()
+})

ui/media/js/utils.js

@@ -1097,6 +1097,48 @@ async function deleteKeys(keyToDelete) {
     }
 }
 
+/**
+ * @param {String} Data URL of the image
+ * @param {Integer} Top left X-coordinate of the crop area
+ * @param {Integer} Top left Y-coordinate of the crop area
+ * @param {Integer} Width of the crop area
+ * @param {Integer} Height of the crop area
+ * @return {String}
+ */
+function cropImageDataUrl(dataUrl, x, y, width, height) {
+    return new Promise((resolve, reject) => {
+        const image = new Image()
+        image.src = dataUrl
+
+        image.onload = () => {
+            const canvas = document.createElement('canvas')
+            canvas.width = width
+            canvas.height = height
+
+            const ctx = canvas.getContext('2d')
+            ctx.drawImage(image, x, y, width, height, 0, 0, width, height)
+
+            const croppedDataUrl = canvas.toDataURL('image/png')
+            resolve(croppedDataUrl)
+        }
+
+        image.onerror = (error) => {
+            reject(error)
+        }
+    })
+}
+
+/**
+ * @param {String} HTML representing a single element
+ * @return {Element}
+ */
+function htmlToElement(html) {
+    var template = document.createElement('template');
+    html = html.trim(); // Never return a text node of whitespace as the result
+    template.innerHTML = html;
+    return template.content.firstChild;
+}
+
 function modalDialogCloseOnBackdropClick(dialog) {
     dialog.addEventListener('mousedown', function (event) {
         // Firefox creates an event with clientX|Y = 0|0 when choosing an <option>.

ui/plugins/ui/image-editor-improvements.plugin.js

@@ -124,35 +124,17 @@
         // Draw the image with centered coordinates
         context.drawImage(imageObj, x, y, this.width, this.height);
 
-        initImagePreview.src = canvas.toDataURL('image/png');
+        let bestWidth = maxCroppedWidth - maxCroppedWidth % IMAGE_STEP_SIZE
+        let bestHeight = maxCroppedHeight - maxCroppedHeight % IMAGE_STEP_SIZE
 
-        // Get the options from widthField and heightField
+        addImageSizeOption(bestWidth)
-        const widthOptions = Array.from(widthField.options).map(option => parseInt(option.value));
+        addImageSizeOption(bestHeight)
-        const heightOptions = Array.from(heightField.options).map(option => parseInt(option.value));
-
-        // Find the closest aspect ratio and closest to original dimensions
-        let bestWidth = widthOptions[0];
-        let bestHeight = heightOptions[0];
-        let minDifference = Math.abs(maxCroppedWidth / maxCroppedHeight - bestWidth / bestHeight);
-        let minDistance = Math.abs(maxCroppedWidth - bestWidth) + Math.abs(maxCroppedHeight - bestHeight);
-
-        for (const width of widthOptions) {
-            for (const height of heightOptions) {
-                const difference = Math.abs(maxCroppedWidth / maxCroppedHeight - width / height);
-                const distance = Math.abs(maxCroppedWidth - width) + Math.abs(maxCroppedHeight - height);
-
-                if (difference < minDifference || (difference === minDifference && distance < minDistance)) {
-                    minDifference = difference;
-                    minDistance = distance;
-                    bestWidth = width;
-                    bestHeight = height;
-                }
-            }
-        }
 
         // Set the width and height to the closest aspect ratio and closest to original dimensions
         widthField.value = bestWidth;
         heightField.value = bestHeight;
+
+        initImagePreview.src = canvas.toDataURL('image/png');
     };
     
 	function handlePaste(e) {

ui/plugins/ui/lora-prompt-parser.plugin.js

@@ -0,0 +1,119 @@
+/*
+    LoRA Prompt Parser 1.0
+    by Patrice
+
+    Copying and pasting a prompt with a LoRA tag will automatically select the corresponding option in the Easy Diffusion dropdown and remove the LoRA tag from the prompt. The LoRA must be already available in the corresponding Easy Diffusion dropdown (this is not a LoRA downloader).
+*/
+(function() {
+    "use strict"
+    
+    promptField.addEventListener('input', function(e) {
+        let loraExtractSetting = document.getElementById("extract_lora_from_prompt")
+        if (!loraExtractSetting.checked) {
+            return
+        }
+
+        const { LoRA, prompt } = extractLoraTags(e.target.value);
+		//console.log('e.target: ' + JSON.stringify(LoRA));
+    
+        if (LoRA !== null && LoRA.length > 0) {
+            promptField.value = prompt.replace(/,+$/, ''); // remove any trailing ,
+    
+            if (testDiffusers?.checked === false) {
+                showToast("LoRA's are only supported with diffusers. Just stripping the LoRA tag from the prompt.")
+            }
+        }
+                                 
+        if (LoRA !== null && LoRA.length > 0 && testDiffusers?.checked) {
+            for (let i = 0; i < LoRA.length; i++) {
+            //if (loraModelField.value !== LoRA[0].lora_model) {
+                // Set the new LoRA value
+				//console.log("Loading info");
+				//console.log(LoRA[0].lora_model_0);
+				//console.log(JSON.stringify(LoRa));
+				
+                let lora = `lora_model_${i}`;
+                let alpha = `lora_alpha_${i}`;
+                let loramodel = document.getElementById(lora);
+                let alphavalue = document.getElementById(alpha);
+				loramodel.setAttribute("data-path", LoRA[i].lora_model_0);
+                loramodel.value = LoRA[i].lora_model_0;
+                alphavalue.value = LoRA[i].lora_alpha_0;
+                if (i != LoRA.length - 1)
+                    createLoraEntry();
+            }
+                //loraAlphaSlider.value = loraAlphaField.value * 100;
+                //TBD.value = LoRA[0].blockweights; // block weights not supported by ED at this time
+            //}
+            showToast("Prompt successfully processed", LoRA[0].lora_model_0);
+			//console.log('LoRa: ' + LoRA[0].lora_model_0);
+			//showToast("Prompt successfully processed", lora_model_0.value);
+			
+        }
+            
+        //promptField.dispatchEvent(new Event('change'));
+    });
+    
+    function isModelAvailable(array, searchString) {
+        const foundItem = array.find(function(item) {
+            item = item.toString().toLowerCase();
+            return item === searchString.toLowerCase()
+        });
+
+        return foundItem || "";
+    }
+
+    // extract LoRA tags from strings
+    function extractLoraTags(prompt) {
+        // Define the regular expression for the tags
+        const regex = /<(?:lora|lyco):([^:>]+)(?::([^:>]*))?(?::([^:>]*))?>/gi
+
+        // Initialize an array to hold the matches
+        let matches = []
+
+        // Iterate over the string, finding matches
+        for (const match of prompt.matchAll(regex)) {
+            const modelFileName = isModelAvailable(modelsCache.options.lora, match[1].trim())
+            if (modelFileName !== "") {
+                // Initialize an object to hold a match
+                let loraTag = {
+                    lora_model_0: modelFileName,
+                }
+				//console.log("Model:" +  modelFileName);
+        
+                // If weight is provided, add it to the loraTag object
+                if (match[2] !== undefined && match[2] !== '') {
+                    loraTag.lora_alpha_0 = parseFloat(match[2].trim())
+                }
+                else
+                {
+                    loraTag.lora_alpha_0 = 0.5
+                }
+				
+        
+                // If blockweights are provided, add them to the loraTag object
+                if (match[3] !== undefined && match[3] !== '') {
+                    loraTag.blockweights = match[3].trim()
+                }
+        
+                // Add the loraTag object to the array of matches
+                matches.push(loraTag);
+				//console.log(JSON.stringify(matches));
+            }
+            else
+            {
+                showToast("LoRA not found: " + match[1].trim(), 5000, true)            
+            }
+        }
+
+        // Clean up the prompt string, e.g. from "apple, banana, <lora:...>, orange, <lora:...>  , pear <lora:...>, <lora:...>" to "apple, banana, orange, pear"
+        let cleanedPrompt = prompt.replace(regex, '').replace(/(\s*,\s*(?=\s*,|$))|(^\s*,\s*)|\s+/g, ' ').trim();
+		//console.log('Matches: ' + JSON.stringify(matches));
+
+        // Return the array of matches and cleaned prompt string
+        return {
+            LoRA: matches,
+            prompt: cleanedPrompt
+        }
+    }
+})()